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We  develop  a numerical  algorithm  for  solving  two-set  graph  partitioning  prob- 
lems. This  algorithm,  based  on  the  gradient  projection  method  and  analysis  of  active 
sets,  exploits  a continuous  quadratic  programming  formulation  of  the  discrete  prob- 
lem. As  a by-product,  we  obtain  an  efficient  scheme  for  projecting  a point  onto  the 
convex  set  K = {0  < x < 1,  lTx  = m}  for  a given  integer  m. 
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CHAPTER  1 
INTRODUCTION 


Graph  partitioning  is  an  important  problem  that  has  extensive  applications 
in  many  areas,  including  scientific  computing,  circuit  board  and  micro-chip  design, 
other  layout  problems  (see  [15]),  and  sparse  matrix  pivoting  to  get  good  ordering 
of  the  unknowns  in  order  to  reduce  the  number  of  nonzero  entries  generated  during 
Gaussian  Elimination.  The  problem  is  to  partition  the  vertices  of  a graph  into  p parts 
such  that  the  number  of  edges  connecting  vertices  in  different  parts  is  minimized.  A 
good  ordering  of  a sparse  matrix  dramatically  reduces  both  the  amount  of  memory 
as  well  as  the  time  required  to  solve  the  system  of  equations(see  [11]).  A class  of 
graph  partitioning  algorithms  reduces  the  size  of  the  graph  ( i.e coarsens  the  graph) 
by  collapsing  vertices  and  edges,  partitions  the  smaller  graph,  and  then  uncoarsens 
it  to  construct  a partition  for  the  original  graph.  These  are  called  multilevel  graph 
partitioning  schemes  (see  [5],  [6]). 

The  2-way  graph  partitioning  problem  is  defined  as  following  : Given  a graph 
G = (V,  E)  with  |U|  = n,  partition  V into  two  subsets,  V\  and  V2,  such  that  Vi  fl  V2  = 
0 for  | Vi  | = m and  | V2 1 = n — m and  the  number  of  edges  of  E whose  incident 
vertices  belong  to  different  subsets  is  minimized.  A 2-way  partition  of  V is  commonly 
represented  by  a partition  vector  P of  length  n,  such  that  for  every  vertex  v € V,  P[u] 
is  0 or  1,  indicating  the  partition  set  to  which  vertex  v belongs.  Given  a partition  P, 
the  number  of  edges  whose  incident  vertices  belong  to  different  subsets  is  called  the 
edge-cut  of  the  partition. 

We  use  a continuous  programming  formulation  for  the  following  graph  par- 
titioning problem  : Given  a graph  with  n vertices  and  a positive  integer  m < ?r, 
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partition  the  vertices  into  two  parts,  one  with  m vertices  and  the  other  with  n — m 
vertices,  where  the  partition  is  chosen  to  minimize  the  number  of  edges  whose  incident 
vertices  belong  to  different  subsets. 

We  test  our  scheme  on  a number  of  graphs  arising  in  linear  programming  and 
G-sets  ( ftp://dollar.biz.uiowa.edu/pub/yyye/Gset/  ).  Our  experiments  show  that 
our  scheme  produces  partitions  that  are  good  enough  to  compete  with  those  produced 
by  the  Metis  program,  and,  in  some  cases,  we  obtain  better  partitions. 

Let  V denote  the  set  ofnxn  permutation  matrices.  Let  G be  a graph  with  n 
vertices  V = {1,2,  • • • ,n},  and  let  £ denote  the  set  of  edges  associated  with  G.  The 
edges  are  ordered  pairs  of  the  form  (i,j)  where  i,j  £ V.  We  assume  that  (z,  i)  <£  £ 
and  if  (i,j)  £ £,  then  (j,  i)  £ £■  The  (i,j)  element  of  the  adjacency  matrix  A0  is  1 if 
( i,j ) £ £,  while  it  is  0 otherwise.  Consider  the  following  two  minimization  problems: 

m n 

E (pTAop)0.  (i.i) 

j = l i=m+l 

and 

min(l  — x)tAx 

subject  to  0 < x < 1,  lTx  = m ' ' ' 

where  1 is  the  vector  whose  entries  are  all  one  and  A = A0  + 1.  Let  lm  be  the  vector 
whose  first  m entries  are  one  and  whose  remaining  entries  are  zero.  We  know  (see 
[7])  that  (1.1)  and  (1.2)  are  equivalent  in  the  following  sense  : 

If  x = Plm,  where  P £ V,  is  a solution  to  (1.2),  then  P is  a solution  to  (1.1). 
And  conversely,  if  P £ V is  a solution  to  (1.1),  then  x = Plm  is  a solution  to  (1.2). 


CHAPTER  2 

GRADIENT  PROJECTION  METHOD  WITH  ACTIVE  SET  STRATEGY 


To  solve 

min(l  — x)tAx 

subject  to  0 < x < 1,  lTx  = m , 
we  will  use  Gradient  Projection  Method  : 

x(q)  = Proj, i-[x‘  - ctV/(x‘)] 
x‘+'  = x(at) 

where  Projx  represents  the  projection  onto  a convex  set  I\ . 

A step-size  a*  can  be  chosen  in  various  ways,  for  example,  Constant  step-size, 
Armijo’s  rule  or  Limited  minimization  method,  etc.  We  will  choose  a such  that 

/(x(afc))  = min/(x(a)). 

a>0 

We  will  focus  on  how  to  evaluate  the  projection, 

x(a)  = ProjK[xk  + ag], 

where  g = — V/(xfc).  That  is, 

min  ^ || x - y ||2 

subject  to  0 < x < 1,  lTx  = m 
where  y is  a given  vector. 

1 Optimality  Condition 


minj||x-y||2 

subject  to  0 < x < 1,  lTx  = m 
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where  y £ Rn  is  given.  The  solution  to  the  problem  (2.1)  is  Projx(y).  Since  ||x  — y||2 
is  a strongly  convex  function,  there  exists  a unique  minimizer  and  the  following  first- 
order  optimality  conditions  hold:  There  exist  A € Rl  and  n,v  € Rn  such  that 


x-y  + /i-^  + Al  = 0, 

(2.2) 

lTx  = m, 

(2.3) 

/rT(l  — x)  = 0,  vTx  — 0, 

(2.4) 

H > 0,  v > 0, 

(2.5) 

0 < x < 1. 

(2.6) 

Conversely,  if  x,/i,^  and  A satisfy  these  conditions,  then  x is  the  unique  minimizer 

of  (2.1). 

For  given  U,L  C {1,2,  ••  • , n},  and  F = (U  U L)c , we  define 

X{  = 1 if  i G U,  Xi  = 0 if  i G L 

(2.7) 

Hi  = 0,  Vi  = 0 if  i € F 

(2.8) 

x (\U\-rn  + J2ieFyi) 
x~  in 

(2.9) 

X{  = yi  — A if  * G F 

(2.10) 

Hi  = Vi  ~ 1 — A if  i 6 77,  Vi  = X — yi  if  i 6 L. 

(2.11) 

From  (2.2),  (2.7),  and  (2.11),  we  have 

Hi  = 0 for  i 6 L and  U{  = 0 for  i € U. 

Since  if  x, //,i/  and  A satisfies  the  conditions,  (2.2)  - (2.6),  x is  the  unique 
minimizer  of  (2.1),  if  we  show  that  the  choices  for  x,/r,  v and  A,  (2.7)  - (2.11)  satisfy 
the  conditions  (2.2)  - (2.6),  then  x in  (2.7)  and  (2.10)  is  the  unique  minimizer  of 

(2.1) .  At  first,  let  us  show  that  these  choices  for  x,^/,^  and  A,  satisfy  the  conditions 

(2.2) ,  (2.3)  and  (2.4)  : 


(2.2).x  -y-f^-F  + Al  =0,  i.e.  xt-  - yi  + m - Vi  + A = 0 Vi. 
For  any  i G U,  xt-  = 1,  //,  = t/,-  — 1 — A and  Vi  = 0.  Hence 
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Xi  - yi  + y,  - Vi  + A = 0. 

For  any  i G L,  Xi  = 0,  f,-  = A — yi  and  yi  = 0.  Hence 

x,-  - yi  + ^ - v>i  + A = 0. 

For  i G F,  yi  = 0,  f,  = 0 and  x,  = yi  — A.  So, 

Xi  - yi  + yi  - k + A = 0. 

Therefore, 

Xi  — yi  + yi  — K + A = 0 Vi. 

(2.3).  lTx  = m,  that  is,  Xl"=i  = m- 

n 

y ' Xi  — itf  i + ^ ) Xi 

i=l  ieF 

= M + £>  - 1*1* 

ieF 

= |r[  + Ea-|F|(|y|-m  + E-^) 

ieF  ^ ^ 

= m 


(2.4).  pT(l  — x)  = 0 and  f;x  = 0,  that  is,  //,(1  — xt)  = 0 and  FjX,  = 0 for  all  i. 

x,  = 1 for  i G U and  yi  = 0 for  i ^ £/,  and  Xi  = 0 for  i G f and  f;  = 0 for 

i ^ L.  Hence, 

//i(l  — x,)  = 0 and  /x,Xi  = 0 Vi. 

We  showed  that  the  choices  for  x,/t,f  and  A,  (2.7)  - (2.11)  satisfy  the  conditions 
(2.2)  - (2.4).  Now,  if  we  show  that  the  choices  for  x,p,F  and  A,  (2.7)  - (2.11)  satisfy 

the  conditions  (2.5)  and  (2.6),  y > 0 and  f > 0,  and  0 < x < 1,  then  the  choices  for 

x,  y,is  and  A,  (2.7)  - (2.11)  satisfy  the  first-order  optimality  conditions  (2.2)  - (2.6), 
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and  x is  the  solution  of  (2.1).  Suppose  that  y(a)  = x0  + «g  where  a > 0 scalar, 
0 < x0  < 1 and  lTXo  = m.  We  now  show  how  to  compute  the  associated  minimizer 
x = x(a)  corresponding  to  y = y(a)  in  (2.1). 

Definition  1 An  inequality  constraint  p,(x)  < 0 is  said  to  be  active  at  a feasible 
point  x if  p,(x)  = 0 and  inactive  at  x if  g,(x)  < 0. 


2 Procedures  to  Determine  Active  Sets 


To  determine  active  sets  for  a = 0+  and  a = a]-,  we  need  the  following  two 
procedures,  Starting  Procedure  and  Continuing  Procedure. 

Finite 


a0  0Cj  a2  a3  • • • a 


■>  a 


For  each  interval,  same  U and  L 

Figure  2.1:  Finite  intervals  of  a 

2.1  Starting  Procedure 

Let  B i = {i  : x0,  = 1},  B0  = {i  : x0i  = 0),  F0  = {1,2,  ••  • ,n),  and  U0  — L0 


0.  Let 


SieF,  9i 

9j  ~ \Fi\  ’ 
7i  = 9i~Tj 

*€Bi\Uj,  gi>gj 


(2.12) 

(2.13) 


and 


Tj  = Y 9j-9i ■ (2-14) 

i€B0\Lj,  si <??7 

Define  Uj+ 1 and  Lj+ 1 in  the  following  way.  For  each  j,  if  7 j > Tj,  then  put  each 
i G B\  \ Uj  such  that  gi  > gj  into  Uj+ 1,  and  if  7 j < Tj,  then  put  each  i G Bo  \ Lj+i 
such  that  51,-  < ~g]  into  LJ+1 . That  is, 

Uj+ 1 = u {*  eBj\  Uj  : gi>g]}  if 

= Uj 


if  7j  > Tj 

otherwise 


(2.15) 


and 


if  7 j < Tj 
otherwise 


(2.16) 


Lj+i 

Lj+i 


Lj  U {i  6 B0  \ Lj  : gt  < g3) 

L 


Fj+ 1 — Fo  \ (pj+ 1 U Lj+i^J  . (2.17) 

Lemma  1 If  jj  > Tj,  then  ~gj  >]jk  for  all  k > j,  and  if  •jj  < Tj,  then  ~gj  <~gu  for  all 
k > j. 


Proof.  First,  let  us  consider  the  case  7 j > Tj.  Let  l be  the  first  index  greater  than 
equal  to  j for  which  7 ;+i  < r;+1.  Hence,  7,  > rt  for  all  j < i < l.  In  (2.15),  we 
remove  the  indices  t E Bx  \ Ui-i  for  which  gt  > g,  from  F%  for  each  i,  j < i < l. 
Thus,  gj  > ■ ■ ■ > gi  > gif~[  and  so,  if  j < k < l + 1 then  g]  > ~gl.  Let  k be  the 
first  index  greater  than  l for  which  7^  > r*,.  Hence,  7 ; < Tt  for  all  7 + 1 < i < k. 
In  (2.16),  we  remove  the  indices  t € B0\  Li- 1 for  which  gt  < g{  from  F,  for  each  i , 
/ + 1 < i < k.  Thus,  gi+i  < • • • < < ~§k-  We  will  show  that  g]  > gi-  Since 

gj  > • • ■ > gj,  it  is  enough  to  show  that  gi  > 7fl  for  k > l + 1.  Note  that  g]  = gl  holds 
if  j = l and  7 j = Tj  and  there  is  no  indices  i 6 B\  \ Ui- 1 such  that  gj  > g%  > g+i- 
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By  way  of  contradiction,  suppose  > gi.  Without  loss  of  generality,  we  can  assume 
9k  >gi>  he. 

9i+\  < • • • < gj~[  < gi  < gi.  (2.18) 


Let  us  define  the  set,  N\  as  the  set  of  indices  in  B\  \ Ut , such  that  gx  > gi,  and  for 
l < q < k,  Mq  as  the  set  of  indices  in  B0  \ Lq  such  that  gi  < 7^.  Then 


9k 


WiW  - Ejv,g»-  ~ SS=?+i 
\h\ 


6 Mq  9i 


(2.19) 


Hence, 

9k -gi 

Since  gk>gi , by  (2.20), 


E,=<+1  9i)  lL/N,(9i  9i) 

\h\ 


k — 1 

X X ~ X^2  _ 9i)  > °- 

q=l+l  i£Mq  Ni 


On  the  other  hand,  since  7 / > 77,  we  have 


(2.20) 


(2.21) 


^2(Si  - 9l)  > _ Si)-  (2.22) 

iVj  Ml 

Hence,  from  (2.21),  we  have 

k—  1 

X X^“s,)"X®“^>0'  (2-23) 

9=/+l  ieM,  M( 

Since  for  l + 1 < <7  < k — 1,  B0  \ Lq  is  monotone  decreasing  and  g^  < gi  (2.18),  we 
have 

Mq  C Mi 

and  so, 

k—  1 

X X^-^<X^-^-  (2-24) 

q=l+ 1 ieMq  Mi 

This  contradicts  (2.23).  Therefore,  gf  > <%  if  A;  satisfies  7,-  < 77  for  all  / + 1 < i < A;. 
Hence  gj  >gii. 
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Now  consider  a general  k > j.  We  will  use  mathematical  induction  to  show 
that  gj  >gf  for  all  k > j.  Let  k{  be  the  i-th  number  after  j such  that  'fk.-i  < 7fc,-i 
and  7 k,  > Tkl.  We  have  seen  that  g~  > g7  for  j < k < kx.  Suppose  it  is  true  for 
j < k < k{.  Then  gj  > g f~-  Let  k{  < k < then  by  the  above  argument,  we  have 
g77  > 9k-  Hence,  ~g]  > g7  for  all  k > j.  Similarly,  we  can  show  that  if  7 j < Tj,  then 
gj  < gL  for  all  k > j.  □ 

Hence,  by  Lemma  1, 


9i  > 9k  Vi  G Uj,  \/k  > j , 


(2.25) 


and 

9%  < V*  € Lj , Vfc  > j.  (2.26) 

Let  us  stop  this  process  if  7,  = 7j  = 0.  Then  since  B0  and  B\  are  finite,  this  must 
be  terminated  in  finite  steps.  Let  s is  the  first  number  such  that  7 s = ts  = 0,  then 
define  U0  = US  = Us+U  L0  = Ls  = Ls+ 1 and  F0  = Fs  = F0  \ {Us  U Ls). 

Thus,  by  definitions  of  UJ+i  and  Lj+ 1,  (2.15),  (2. 16), and  (2.25),  (2.26), 

X/ie/L  V*  _ ^z'cFq  V;  -r 


0*  > 


l*b| 


if  k (E  Uq  — Us, , 


and 


„ ^ 9i  _ Vi  :c  i„  ^ r t 

9k  < ^77;  — — rrvj — if  k £ Lq  — Ls. 


2.2  Continuing  Procedure 


I*1. 


I*b| 


(2.27) 


(2.28) 


min  ||x  — y ||2 

s.t.  0 < x < 1,  lrx  = m 

where  1T  = (1, 1,  • • • , 1)  and  y = x*  -f  ag.  Let 

Uj  = U{atj)  = {i  : Xi(aj)  = 1}  , Lj  = L(ctj)  = {i  : 27(07)  = 0}  . 
Lemma  2 If  [/(an)  — U(a2)  and  L{ati)  — L(a2),  then  for  all  ot\  < a < a2) 

U(a)  = U(qi)  and  L(a)  = L(ai). 
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Proof.  Suppose  that  U(a.\)  = U(a2)  and  L(a j)  = L(a2).  To  show  that  U(a)  = 
U(a\)  and  L(a)  = L(ct\)  for  all  «i  < a < a2,  we  need  to  show  that  for  0 < s < 1, 


x(Qi  + s(a2  - ai))  = x(ax) + s(x(a2) -x(ai)),  (2.29) 

/j.{qi+  s(a2  - ai))  = /z(ai)  + s(^(a2)  - /i(ax)),  (2.30) 

j/(ai  + s(a2  - c*i))  = v(ax)  + s(z/(a2)  - ^(on))  (2.31) 

and 

A(ai  + s(a2  — ai))  = A(ax)  + s(A(a2)  — A(ai)).  (2.32) 


Suppose  that  x,  and  A satisfy  the  conditions  (2.2)  — (2.6),  then  x(ai  + s(a2  — 
ai)),  /x(qi  + s(a2  — Qi)),  v(ai  + s(a2  — au)),  and  A(ax  + s(a2  — ax))  satisfy  the 
conditions  (2.2)  — (2.6). 

If  we  show  that  x(ai) + s(x(a2)  - x(ax)),  /u(qi)  + 5(/u(q2) -/r(ai)),  u(ai)  + s(v(a2)  - 
t'(ai)),  A(«i)  + s(A(q2)  — A(ax))  satisfy  the  conditions  (2.2)  — (2.6),  then  by  the 
uniqueness  of  the  optimality  condition,  we  have  (2.29),  (2.30),  (2.31),  (2.32). 

The  condition  (2.2)  is  satisfied  as  follows  : 

xt(ai)  + 5(xt(a2)  - £;(«!))  - (out/;  + s(a2yt  - ax y{)) 

+ (Ma l)  + s(/j.i(a2)  - m{a i)))  + (^i(ai)  + s(i/{(a2)  - z/,(ai))) 

+ A(aj)  -f  s(A(a2)  — A(qj)) 

= (1  - •§)  [aq(ai)  - any,  + m(ai)  + ^(aq)  + A(ax)] 

+ s [x;(a2)  — a2y,  + y,(a2)  + ^(a2)  + A(a2)] 

= 0 

since  X;(a2)  — ayy;  + yi(ctj)  + + A (aj)  = 0 for  all  i and  j = 1,2. 

The  condition  (2.3)  is  satisfied  as  follows  : 

1T  [x(ax)  + s(x(a2)  - x(oi))]  = lrx(ai)  + s (lTx(a2)  - lrx(ax)) 

= m + s(m  — m) 


m. 
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The  condition  (2.4)  is  satisfied  as  follows  : 

[M«i)  + - ^(oi))]r  [1  - x(ax)  + s(x(a2)  - x(a-i))]  = 0 

& lMi(ai)  + s(fj,i(a2)  - /u:(ai))]  [1  ~ Zi(ai)  + s(xi(a2)  - x;(ai))]  = 0 Vi.  (2.33) 

For  each  i,  if  i € U(ax)  = U(a2),  then  x;(ai)  = Xi{a2)  = 1,  and  if  i ^ U(ax)  = U(a2), 
then  //,-(ai ) + s(fii(a2)  — ^,-(ai))  = 0.  Hence,  (2.33)  is  satisfied. 

Kqi)  + SM<*2)  - ^(«i))]T  [x(ai)  + s(x(a2)  - x(ax))]  = 0 

& [yi(a i)  + s(ui(a2)  - ^(ai))]  [xi(ai)  + s(x,(a2)  - x,^^))]  = 0 Vi.  (2.34) 

For  each  i,  if  i G L(a i)  = L(a2),  then  x,(ai)  = x,-(a2)  = 0,  and  if  i ^ L(ax)  — L(a2), 
then  Ui(ai)  T s(z/,(a2)  — Ui(ax))  = 0.  Hence,  (2.34)  is  satisfied.  Therefore,  the 
condition  (2.4)  is  satisfied. 

We  now  show  that  the  condition  (2.5)  is  satisfied  as  follows  : 

/r(ai)  + s(/i(a2)  - /i(«i))  > 0 

O (1  - s)n(ai)  + sfi(a2)  > 0.  (2.35) 

Since  n(ax)  > 0,  /i( a2 ) > 0 and  0 < s < 1,  (2.35)  holds.  Also  observe  that 

v(ax)  + s(is(a2)  - v(ai))  > 0 

O (1  — s)v(cti)  + su(a2)  > 0.  (2.36) 

Since  z/(q i)  > 0,  z/(a2)  > 0 and  0 < s < 1,  (2.36)  holds.  Hence,  the  condition  (2.5) 
is  satisfied. 

The  condition  (2.6)  is  satisfied  as  follows  : 

0 < x(ai)  + s(x(a2)  - x(at))  < 1 
O 0 < x,(ai)  + 5(xj(a2)  - Xj(ai))  < 1 Vi 
O 0 < (1  — s)xj(ai)  + sx{(a2)  < 1 Vi.  (2.37) 
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Since  0 < a:,(ai),  x,(a2)  <1—5,  0 < (1  — 5)xj(q1)  <1—5  and  0 < sx^cc?)  < 5. 
Hence,  (2.37)  satisfied  and  so,  (2.6)  is  satisfied.  We  showed  that 

*(<*i)  + s(x(a2)  - x(ai)),/x(aj)  + s(/i(a2)  - /i(aq)), i/(ai)  + s(^(a2)  - z/(an)) 

and 

A(c*i)  + s(A(q2)  — A(aj)). 

satify  the  conditions  (2.2)  — (2.6),  and  so,  by  the  uniqueness  of  the  optimality  condi- 
tion, (2.29)  — (2.32)  hold.  Thus,  for  all  cc\  < a < a2, 

U(a)  = U{ot\)  and  L(a)  = L(ai). 


□ 

By  Lemma  2,  we  can  conclude  that  there  are  only  finite  number  of  breaks. 
Define  U(atj+ 1)  and  £(aJ+1)  such  that 


if  Hi(aJ+i)  = 0 and  Xi(aj+i ) = 1 , then  i G 7/(aJ+1),  (2.38) 


and 


if  Vi(aJ+i)  = 0 and  Xj(oj+i)  = 0,  then  i G £(aJ+i).  (2.39) 


Let 


F°  = Fj[JU(aj+1)\JC(aj+1),  U°  = 0 and  L°  = 0. 


For  0 < /,  define 


~l  _ 9i 

9 |F'|  ’ 

T = E _ (a  “ S') 

ieU(aj+l)\Ul,gi>gt 


and 


E _ (/  - v)  ■ 

ieC(aJ+1)\Ll,gi<g‘ 


(2.40) 


(2.41) 

(2.42) 


Ti  = 


(2.43) 
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Also,  define 


ul+1  =Ul  U | i G U(aj+1)  \ U‘  : gt  > g'j  if  7 7 > 77 
Ul+1  = Ul  otherwise  , 

Ll+1  = Ll  U € £i(aJ+i)  \ Ll  : # < #'}  if  7/  < t, 
Ll+1  — Ll  otherwise 


(2.44) 

(2.45) 


and 


Fl+1  = F°\  (Ul+1  U Ll+1) . (2.46) 

Let  us  stop  this  process  if  7 7 = 77  = 0.  Apply  Lemmal  with  Bx  — U(ctj+1),  B0  = 
£{aij+ 1),  F0  = F°  and  7 j+i  = 7/,Tj+i  = 77.  Then  since  77(aj+i)  and  C(aJ+x)  are  finite, 
this  must  be  stopped  in  finite  steps.  Let  s be  the  first  number  such  that  7 s = ts  — 0, 
then  define  Uj+ 1 and  Lj+X  with  FJ+X  = ( Uj+i  U Lj+ x)c  as  follow  ; 


Uj+1  = (Uj\U(aj+1))UUs 


and 


Lj+ 1 - ( Lj  \ £(qj+1))  U Ls. 
Then  as  similar  to  2.27  and  2.28, 


(2.47) 


L48) 


9k 


and 


9k  < 


> ^JJ+\ 9l  if  k € Us 

r j+i  I 
YhieF]Jtl  9i 


l^f+i  I 


if  k e Ls. 


(2.49) 


(2.50) 


We  now  define  functions  x(Fj,  ay),  g(Fj,  ctj),  A(Fj,  aj),  with  0 = a0  < Qi  < a2  ■ ■ 
where  (Fj+X,  aj+1)  are  obtained  from  (Fj,ctj)  in  the  following  way  ; 


w , (|7/,|  ~ m + ^ieF  yi{a)) 

A(q)  = — , if  aj  <a  < aj+ 1 

Iwl 

_ \Uj\  — m J2ieF,  xo,  a I^ieFj  9i 


IF, 


\F3 


l*il 


Xi(a)  — yi(a)  — A (Fj,a),  if  i G Fj  and  ctj  < a < aJ+1 


(2.51) 


(2.52) 


/ J2ieF]  9i  ) 

1 1 Uj 

| - 

m HieFj 

®o, 

r \fa  ) 

1 

l*il 

l*)l 
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lu-i(a)  = yi(a)  — 1 — A (Fj,a),  if  i G Uj  and  ctj  < a < aJ+i  (2.53) 

( \Uj\-m  J2ieF,xo. 

x°'  if i )-— ifi  iAi- ’ 

14(a)  = X(Fj,a)  — yi(a),  if  * € Lj  and  a,  < a < aJ+1  (2.54) 

F I - (HisFjSi 

+ — 1-  a I — — 9.  | - *o. 


IFI 


I FI 


IFI 


and 


sup{o  : pt(a)  > 0 \/i  G Uj,m(ap  / 0}, 

CKj+i  = min  <(  sup{a  : vt(a)  > 0 Vi  G Fj,  F,(a+)  ± 0}, 

supja  : 0 < Xi(Fj,a)  < 1 Vi  G Fj,x,(o;+)  ^ 0,  1} 


(2.55) 


By  (2.2),  (2.52),  (2.53),  (2.54)  and  (2.55),  we  have 


Pi(cn)  — 0 for  i $ Uj,aj  < a < Oj+i, 


(2.56) 


Uj(a)  = 0 for  i Lj,aj  < a < aJ+1, 
x,(a)  = 1 for  i G Fj,aj  < a < Qj+j 


and 


xfa)  = 0 for  i G Z/y , ory  < a < ay+1. 


(2.57) 

(2.58) 


(2.59) 


Theorem  3 For  all  i and  a,  x,-(a),  A(a), pi(cx)  and  Vi(cx)  are  continuous  piecewise 
linear  functions  of  a,  and  0 < xfa)  < 1. 


Proof.  We  want  to  show  that  x(Fj,  a),  A(Fj,  a)  and  p(Fj,a)  are  continuous  and 
peicewise  linear.  Suppose  A is  the  set  of  indices  i such  that  x,-  = 1 and  D is  the 

set  of  indices  i such  that  xt-  = 0 where  A = 4i  U d2  and  D = D\  U D2  with 

A\  — Uj  n Fj_i,  A2  = Uj— i n Fj,  D\  = Lj  n Fj_ i,  D2  = Lj-i  fl  Fj.  Then 

Wi\  = I ZTy—i | + |Ai|  - |A2|,  \Lj\  = |Vi|  -f  \Di\  — \D2\ 


and 


|Fj|  = IFj-il  — (|Ai|  — |A2|  + |Fr|  — |F2|). 
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Also,  by  definition  of  Xi(Fj,a),( 2.52), 


Vi  — 1 + Qj),  if  i G A 


and 

Vi  = A(Fj_i,Qj),  if  i € D. 


Hence, 
A(^i-i  i &j) 


l^-il 

\Uj-\ \ — m + Yhi^Fj  y*  + Sie^i  y * ~ F ”1"  SieD!  2/«  ~ Yhi^D2  F 

|Hj|  — m + Y^ieFj  y*  + ^te(Ai\A2)u(Ci\D2)  A(Fj-1,aj) 

l^-il 


Since  (|F,_i|  - (|Ai|  - |A2|  + |£h|  - |F2|))  A(Fj_i,aj)  = |C/j|  - m + XlteF,  2/*  > 

wz?  ^ _ 1^1  ~ m + £,-€f,  y. 

1 l^-il  - (|Ax|  -|A2|  + IAI-P2I) 

|[/,|  - m + E.gf,  y« 

1^1 

= A(FJ,aj). 


Thus,  by  definitions  of  x,(a)(2.52),  ^,(a)(2.53)  and  i/,-(a)(2.54), 


Xi(Fj-i,aij ) = £;(Fj,  aj)  Vi  and  j, 

PiiFj-iiOtj)  = a,-)  Vi  and  j, 

and 

Vi{Fj- 1 , Oj ) = i/,-(  Fj , a j ) Vi  and  j. 

Hence,  A(a),  x(a)/r(a)  and  f(q)  are  continuous.  Moreover,  since  x(a),  A (a),/u(a)  and 
v(a)  are  piecewise  linear,  derivatives  exist  on  the  interior  of  any  interval  (aj,aj+1). 

Now  we  want  to  show  that  0 < x;(a)  < 1 for  all  i that  is,  0 < x(a)  < 1 for 
all  a > 0.  If  we  show  that  x,(0+)  < 1 for  x0i . = 1,  and  x,(0+)  > 0 for  x0,  = 0,  i.e.  for 
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k G Fo  fl  B\,  x^(0+)  < 1 and  for  k G Fo  fl  Fo,  x*:(0+)  > 0,  then  by  continuity  of  x(a), 
and  definition  of  x(a)  and  aJ+1,  (2.52),  (2.55),  (2.58),  and  (2.59),  0 < xt-(a)  < 1 for 
all  i and  a > 0. 

Suppose  that  k G Fo  fl  B\.  Then  since  7 / = 0,  by  definition  of  -yy (2.13),  we 
have  {i  G Fi  \ F/  : g,  > gj}  = 0,  i.e., 


^ — ^-'t'6F|  9t  X/ieFo 

“ ThT  = 


Vfc  G {B 1 \ U{)  = F0OB1 


(2.60) 


Hence, 


Since  Xq*  = 1,  therefore, 


Zfc(0+)  = gk 


Xy  igFp  9i  „ 

\F0\ 


xk{Fo,0+)  = 1+  0+ 


< 1. 


Suppose  H F0  fl  F0.  Then  since  77  = 0,  by  definition  of  7j(2.14),  we  have  {i  G 
B0  \ U : gi  < 57}  = 0,  i.e. 


_ Z^ieFi  9i  l^ieFo  9i 

9k  >gi  = — , - = — 

|F,|  \F0\ 


VA:  G (Fo  \ Li)  — Fq  fl  Bq. 


(2.61) 


Hence, 


Since  Xok  = 0,  therefore, 


^L(0+)  = gk 


X/;GFq  9i  . „ 

|fi>|  - 


^(Fo,0+)  = 0 + 0+ 


> 0. 


Hence,  by  linearity  of  x(a),  for  all  a > 0 , xjt(a)  < 1 for  k G F0  fl  Bx  and  xfc(a)  > 0 
for  k G Fo  fl  Bo-  Therefore,  for  all  i and  a > 0,0  < x,-(a)  < 1.  Thus  0 < x(q)  < 1 
for  all  a > 0.  □ 

So,  the  condition  (2.6)  is  satisfied.  Now  if  we  hold  the  condition  (2.5),  then 
x(a)  is  the  solution  of  (2.1).  First,  let  us  show  that  /z,( 0)  = 0 and  f,(0)  = 0 for  all  i. 
Let  F/  = (Fi  U B0)c  where  Bx  = {i  \ x0|  = 1},  F0  = {i  : x0,  = 0}. 
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If  B\  = 0,  done  for  Bx.  Suppose  Bx  / 0,  then  for  k E Bx, 

/n\  , | n ( ^2ieFj9i\  \Bx\  — m J2ieF,x o, 

where  xtJl  = 1,  and  - = 0 since  Y,*f,  x°-  + Bi  + Bo  = 
YjieF,  xo,  + EieBj  xo,  = ELi  xo,  = m.  Hence,  ^*(0)  = 0 for  all  k E Bx. 

If  Bq  = 0,  done  for  B0.  Suppose  B0  ^ 0,  then  for  k 6 Ho, 

,n\  \Bx\-m  EieF/xo,  (Yjt^F,9i  \ 

•'iio)  = -wr+~m~+0'{  ~w~9y_Io‘ 

where  x0k  = 0,  and  + ^'^Fi\X°'  = 0-  Hence  ^(0)  = 0 for  all  k E B0. 

By  definition  of  p in  F (2.8),  pt(0)  = 0 and  i'i(a)  = 0 for  i E Fi.  Hence,  we 

have 

^,(0)  = 0 and  i/,-(0)  = 0 Vi. 

Now,  we  want  to  show  that  fi(a)  > 0 and  u(a)  > 0 for  all  a > 0.  We  know  that 
Hi(otj)  = 0 Vi  E Fj  = ( Uj  U Lj)c . It  is  sufficient  to  show  that  fii(a)  > 0 Vi  € U3  and 
Ui(a)  > 0,  for  all  i E Lj,  a E (aj,aJ+i)  and  j > 0.  By  definition  of  ctj+ x,  for  all 
a E (aj?aj+i), 


Pi(a)  > 0 Vi  E U3  \ U(aJ+x)  and  Uj(a)  > 0 Vi  E Lj  \ £(aJ+1). 
Hence  by  definition  of  UJ+X  and  Lj+X,  (2.47)  (2.48),  it  is  enough  to  show  that 


p,(a)  > 0 Vi  E Us  and  Ui{a)  > 0 Vi  E Ls  Va  E (aj,otj+ 1). 


Since  /r  and  ^ are  linear  for  the  inteval  (aj,  aJ+i)Vj  > 0,  it  is  sufficient  to  show  that 
for  all  j,  Hi(ct'j)  > 0 Vi  E and  ) > 0 Vi  E Ls  for  all  j. 

From  the  definitions  of  p;(a)  and  z/,(a)  (2.53),  (2.54),  we  have 


- 


Et'eFj 

i^r 


IEI 


- gk- 


and 
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By  (2.27)  and  (2.28),  we  have  p'k{0+)  > 0 and  u'k(0+)  > 0,  and  by  (2.49)  and  (2.50), 
p'k(ot1f ) > 0 for  all  k 6 Us  and  ^(a* ) > 0 for  all  k £ Ls  for  all  j.  Therefore,  pi(a)  > 0 
and  Vi(a)  > 0,  for  all  a > 0 and  i.  This  holds  the  condition  (2.5).  Hence  x(a)  is  the 
unique  minimizer  of  (2.1). 

3 Extreme  Point  of  Convex  Set  K 

Definition  2 A point  x in  a convex  set  C is  said  to  be  an  extreme  point  of  C if  there 
are  no  two  distinct  points  X!  and  x2  in  C such  that  x = oxi  + (1  — n)x2  for  some 
a,  0 < a < 1. 

For  K = {0  < x < 1,  lTx  = m},  by  the  definition  of  extreme  point  of  a 
convex  set,  a point  x such  that  Xi  = 0 or  1 for  all  i is  an  extreme  point  of  K . 

Lemma  4 Let  b and  c be  the  distinct  extreme  points  of  K next  to  an  extreme  point 
of  K,  a.  Then  a,  b and  c are  a vector  where  m components  are  ones  and  n — m 
components,  are  zeros.  Let  6 be  the  angle  between  (a  — b)  and  (a  — c).  Then  9leq9 0°. 
And  so,  for  each  extreme  point  of  the  convex  set  K,  the  6 is  less  than  or  equal  to  90°. 


Proof.  Let  a be  an  extreme  point  of  K and  b and  c be  the  distinct  extreme  points 
of  K next  to  a.  Then 


cos  6 = 


(a  - b)T  (a-c) 


a 


a — c 


(2.62) 


Since  || a — b||  ||a  — c||  > 0,  if  (a  — b)T  (a  — c)  > 0,  then  cos  9 > 0 and  so  9 < 90°. 


(a  - b)T  (a-c)  = 

m ~ I {j  ■ aj  = l,bj  = 1}|  - | {j  : a:  = 1 ,Cj  = 1}|  + |{j  : b,  = l,c;  = 1}|.  (2.63) 

Since  0 : aj  = Cj  = 1}  = {j  : a3  = Cj  = bj  = l}U  {j  : a3  = Cj  — 1,  bj  = 0}  and 
{j  : b3  = cj  = 1}  = {j  : a;  = 0,  bj  = Cj  = 1}  U 0 : aj  = b j = c,-  = 1}, 


(2.63)  = m-(|{j  : aj  = l,bj  = 1}|  + |{j  : a,- = l,Cj  = 1,  bj  = 0}|) 

+ |0  : aj  = 0,  bj  = l,Cj  = 1}|.  (2.64) 
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Since  {j  : ay  = 1,  b;  = 0}  = {j  : ay  = cy  — 1,  by  = 0}u{j  : ay  = 1,  Cy  = 0,  b;  = 0}, 

(2.64)  = m - (|{j  : a3  = l,by  = 1}|  + \{j  : ay  = 1,  by  = 0}|) 

+ l{ j ■ ay  = 1,  by  = 0,cy  = 0}|  + | {j  : ay  = 0,  by  = l,cy  = 1}| 

= m-  (|{j  : ay  = l,by  = 1}|  + 10  : ay  = 1,  by  = 0}|) 

+ 10'  : ay  / bj  = Cy}| 

= m - |0  : ay  = 1}|  + |0  : ay  / by  = cy}| 

= 10  : ay  # by  = Cy}| 

0. 


Hence  6 < 90°.  Since  b and  c were  arbitrary  for  arbitrary  fixed  a,  we  conclude  that 
for  each  extreme  point  of  K , the  6 is  less  than  or  equal  to  90°.  □ 

Let  x is  an  extreme  point  of  K , then  the  normal  cone  to  K at  x is 

Nk(x)  = {p  e X*  : < p,x  - x ><  0 Vx  € K}.  (2.65) 

Lemma  5 For  every  extreme  point  of  the  convex  set  I\ , if  the  6 is  less  than  or  equal 
to  90°,  then  I\  — x C — /Va-(x). 

Proof.  Let  w £ I\  — x,  then  w = v — x,  v € K.  Since  the  angle  of  a convex  set  K 
is  less  than  or  equal  to  90°,  for  all  v and  x € K, 

<v  — x, x — x>  >0. 

That  is, 

< w,  x — x > > 0 V w £ A’-x. 

Hence,  K — x C □ 

Let  us  define  the  projection  from  R"  on  K , as  Projh'(z)  = niinxeA'  h(x)  where 
h(x)  = |||x  — z || 2 . Then  Proj^i z)  = x,  if  and  only  if  Vfi(x»)  • (x  — x*)  > 0 Vx  G K. 
That  is, 

< x,  — z,  x — x»  > > 0 Vx  G K.  (2.66) 
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Lemma  6 Let  x be  an  extreme  point  of  I\  . Then  Projx(z)  = x if  and  only  if 
z - x 6 Na-(x). 

Proof.  Let  Projn( z)  = x.  Then  by  (2.66), 

< x — z,  x - x > > 0 Vx  £ K . (2.67) 

< z - x,  x — x > < 0 Vx  £ A'.  (2.68) 

Therefore, 

z — x £ Nk(5l).  (2.69) 

Conversely,  if  z-x  £ Afc(x)  then  by  definition  of  Nk(x),  (2.65),  we  get  the  condition 
(2.66).  Hence,  Projx(z)  = x.  □ 

Now,  we  need  to  show  that  if  the  projection  reaches  the  extreme  point  with 
a,  then  for  any  a > a,  the  projection  will  be  stay  at  the  extreme  pointof  K. 

Theorem  7 //x  is  an  extreme  point  of  K and  ProjK^o  + ay)  = x where  I\  — x C 
— A^a(x)  and  x0  £ K,  then  Proj^f^o  + ay)  = x for  all  a > a. 

Proof.  Since  K — x C — jVa(x), 

-(x0  - x)  £ N a' (x) . (2.70) 

Since  ProjK'i'X-o  + ay)  = x,  by  lemma  6, 

x0  + ay  - x = z - x £ NK(x).  (2.71) 

where  z = x0  + ay.  We  claim  that  x0  + (a  + S)y  — x.  = z + Sy  — x£  A1at(x).  By 
(2.70),  (2.71)  and  the  definition  of  Abc(x),  (2.65)  we  can  get 

^(-(xo  - x))  + i(z  - x)  = ^oy  £ Nk(x).  (2.72) 

This  implies  that  all  multiple  of  y is  in  NK(x)  i.e.  Sy  £ Nk(x).  Hence,  by  definition 
of  normal  cone  to  K at  x,  (2.65)  and  (2.71), 


x0  T (a  T £)y  - x £ Nh-(x). 
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That  is, 

x0  + ay  - x 6 Afc(x)  Vo  > a. 

Therefore,  by  Lemma  6,  Projxfx-0  + oy)  = x for  all  a > a.  □ 

Hence,  we  conclude  that  if  the  projection  on  the  convex  set  K reaches  an 
extreme  point  of  I\  with  o,  then  the  projection  on  the  convex  set  K will  stay  there 
for  all  a > a.  Therefore,  if  we  reach  the  extreme  point  of  the  convex  set  K,  stop 
marching  on  a. 

4 Stopping  Criteria  for  a Given  Search  Direction 

Definition  3 Let  K be  the  convex  set  defined  by  K = {0  < x < 1,  l7  x = m}.  A 
point  x(aj)  is  the  stopping  point  of  g on  K if  we  have  aJ+i  = +oo. 


Theorem  8 Let  g be  a given  direction  vector  and  Uj,  Lj  be  the  upper  and  lower  active 
sets  and  Fj  inactive  set  at  a — ctj.  Let  Fj  7^  0.  Then  g ,■  > g Vi  G Uj,  gt  < g Vi  G L: 

9 

and  gi  = g\/i£  Fj  where  g = -p  p , if  and  only  if  aJ+1  = +00. 


Proof. 


Since  Fj  7^  0,  there  exist  g — 
gi  = g Vi  G Fj,a  > a3 


12  F,  3% 

\P\  ■ 

xfa ) = gi  — g = 0 V*  G Fj,  a > ctj 
x,(a)  = Xi(aj)  Vi  G F},  a > ctj 


So,  if  0 < Xi(aj)  < 1,  then  0 < x,(a)  < 1 for  all  i G Fj,a  > ctj.  Hence, 

sup{a  : 0 < Xi(Fj,a)  < 1 Vi  G Fj,Xi(Fj,af)  7^  0 or  1}  = +00. 


gi  > g Vi  G Uj,a  > ctj  <*=*>  /r,(a)  = gi  — g > 0 Vi  G Uj,a  > ctj 

O pt{a)  > Pi(otj)  Vi  G Uj,  a > ctj 


So,  if  gi(ctj)  > 0,  then  pi(a)  > 0 for  all  i G Uj,a  > otj.  Thus, 


sup{o  : m(a)  > 0 Vi  G Uj,pi(Fj,aJ)  7^  0}  = -foo. 
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Similarly, 


Qi  < g Vi  G Lj  sup{a  : tsfa)  > 0 Vi  € Lj,Ui(Fj , at)  ^ 0}  = +oo. 


Therefore,  by  the  definition  of  aJ+1,  gi  > g Vi  G Fj,  gi  < g\H  G L}  and  gi  = g Vi  G Fj 
where  5 = ^7 , ■ , if  and  only  if  aJ+1  = +00.  □ 


Lemma  9 Fj  can  not  be  an  empty  set  for  all  j. 


Proof.  Suppose  that  j is  the  first  number  such  that  Fj  = 0.  Then  Fj_  1 7^  0. 

If  Xi  = 1 or  0 Vi  G Fj_i,  then  stop  marching  on  a and  so  Fj  does  not  exist. 
Hence  0 < X{  < 1 for  some  i’s  in  Fj_x  and  x;(aj)  = 1 or  0 Vi  G Fj. 

Recall  the  definitions  in  continuing  procedure  (2.38)  - (2.48).  By  (2.44)  and 
(2.45),  either  Ul  — Ul~l  or  Ll  = Ll~l  for  all  /.  Since  Fj  = 0,  there  exist  s such 


that  Fs 


and  Fs  1 7^  0.  Then  either  Us  = Us  1 or  Ls  = Ls  1 . Without  loss  of 


generality,  assume  Ls  = Ls  1 . Then  Us  \ US  1 = Fs  T Thus,  gt  > gs_1  Vi  G Fs_1 
where  gs_1  — FS~ 1 9l 


■ It  is  a contradiction.  Therefore  Fj  7^  0. 

52 9t  |t/|— m+53t eF  x°i 

By  this  lemma  9,  g — ^ and  — 2 — - are  well  defined  for  all  j. 


□ 


IF 


Theorem  10  For  a given  search  direction  vector  g,  we  must  stop  marching  on  a. 
That  is,  we  reach  the  stopping  point  of  g on  I\  or  the  extreme  point  of  the  convex  set 
K. 


Proof.  If  we  reach  stopping  point,  we  stop  marching  on  a.  Since  n < 00,  by  Lemma 
2,  Theorem8  and  lemma  9,  we  reach  the  extreme  point  of  K if  we  do  not  have  the 
stopping  point  of  g on  K . By  Theorm  7,  we  stop  marching  on  a.  □ 


CHAPTER  3 

EXCHANGE  THE  SUBSETS  FROM  PARTITIONED  GRAPH 


After  partitioning  a graph  into  two  sets,  V and  W,  we  can  improve  by  ex- 
changing some  components  of  V and  those  of  W.  That  is,  we  can  reduce  the  edge-cut 
between  V and  W by  exchanging  some  elements  of  V and  those  of  W.  In  1970, 
Kernighan  and  Lin  [13]  introduced  this  exchange  method  between  two  sets.  The 
Kernighan-Lin  algorithm  uses  pair-swapping,  exchanging  two  components,  one  from 
each  sets. 

We  develop  the  generalization  of  the  Kernighan-Lin  method  from  pair-swapping 
to  set-swapping.  At  first,  we  change  the  graph  partitioning  problem  to  continuous 
problem  (see  [7]).  For  any  m which  is  less  than  min(|K|,  |VU|),  we  exchange  a set  which 
has  m elements  of  V and  a set  which  has  m elements  of  W.  Figure  3.1  gives  us  the 
basic  idea,  and  the  following  theorem  is  the  quadratic  problem  for  the  set-swapping 
method. 

Theorem  11 

min(l  - x1)TAnXi  + (1  - x2)TA22x2  - (1  - x2)rA21Xi  - (1  - x1)TA12x2 
subject  to  0<Xi<l,0<x2<l,  l7Xi  = m = lrx2  , 

has  a 0/1  solution , where  1 is  one  vector  whose  size  is  same  as  that  of  x,-  for  i = 1,2. 

Let  V and  W be  the  nodes  corresponding  to  the  columns  of  An  and  A22  re- 
spectively. Let 

v = V7  u u2,  vi  n v2  = 0,  w = wlu  w2,  wx  n w2  = 0. 

where  Vi  = {i  : xu  = 1},  Wx  = {i  : x2t  = 1}  and  V U W = {1,2,  •••,«}.  Then 
interchanging  Vx  and  \\\  gives  the  biggest  possible  reduction  in  the  edge-cut  generated 
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24 


V w 


E 1 is  the  number  of  edge  cut  between  V 1 and  V 2 

E 2 is  the  number  of  edge  cut  between  W 1 and  W 2 

E 3 is  the  number  of  edge  cut  between  V 1 and  W2 

E 4 is  the  number  of  edge  cut  between  V2  and  Wi 


Figure  3.1:  Partition  the  subgraphs  and  generate  new  partition  by  exchanging  the 
partitions  of  the  subgraphs 
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by  swapping  sets  of  size  to.  Note  : The  minimum  in  the  Quadratic  programming  may 
be  greater  than  or  equal  to  zero  in  which  case  no  improvement  is  possible. 


Use  Gradient  Projection  Method  : Let  x = ( ^ ] . Then 


x(a)  = ProjKxK[xk  ~ aV/(xfc)], 


rk+ 1 


x(a-fc) 


where  Projx  represents  the  projection  onto  a convex  set  K . The  projection  is 


Xi(a)  = ProjK[x\  - agi], 

x2(a)  = ProjK[x * - ag2], 
where  gj  = -VXl/(xfc)  and  g2  = -VX2/(xfc).  That  is, 
min  | [ 1 1 x i - yxll2  + ||x2  - y2||2] 

subject  to  0 < Xi  < 1,  0 < x2  < 1,  lTXx  = m = lTXj 

where  yi  and  y2  are  given  vectors. 

Let  g = ^ j , where 

gi  = -VXl/(xfc) 

— — [Aul  — 2AnXi  — Ai21  + A12x2  + A12x2] 

= An(2xi  — 1)  + A12(l  — 2x2) 
g2  = -VX2/(xfc) 

= — [A22l  — 2A22x2  + A2iXi  — A2il  + A2iXi] 

— A22(2x2  — 1)  + A21(l  — 2xj). 

1 Optimality  Condition  for  Two  Partitioned  Sets 


min  | [||Xl  — yx||2  + ||x2  — y2||2] 

subject  to  0 < Xi  < 1,  0 < x2  < 1,  lTXi  = m = lTx2 


(3.1) 
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where  yi  and  y2  are  given.  That  is 

min  |||x  — y ||2 

subject  to  0 < x < 1,  lTx  = m 


where 


yi 

y2 


y = | :/  ] € Rn  and  x = ( **  ] £ Rn. 


The  solution  to  the  problem  (3.1)  isProjA'xft'(y),  be., 

f ProjK(y1 ) \ 

V ProjK{y2)  ) ' 

Since  ||xi  — y 1 1|2  + ||x2  — y2||2  is  strongly  convex  function,  there  exists  a unique  min- 
imizer  and  the  following  first-order  optimality  conditions  hold:  There  exist  Ai,A2  € 
R 1 and  pi,  /r2,  u\,  z/2  G Rn  such  that 


x-y  + /i-i/  + Al  = 0, 


where  x = ( **  ) , y 


1 0 
0 1 


That  is, 


l7  x = m, 


fiT(  1 — x)  = 0,  vTx  = 0, 


/i  > 0,  v > 0, 


yi 

y2 


0 < x < 1, 


Mi 

M2 


, ^ = 


Vi 

V2 


x<  — y<  + /q  — iq  + A(1  — 0,  2=1,2 
lTxt  = m,  2 = 1,2 
Mf(!  ~xt)  = 0,  ujxt  =0,  t = 1,2 

/q  > 0.  i/t  > 0,  t = 1, 2 


AT  = 


Ai 

A2 


, 1 = 

(3.2) 

(3.3) 

(3.4) 

(3.5) 


0 < xt  < 1,  t = 1,2 


(3.6) 
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Conversely,  if  ~x.t,yt,Vt  and  Xt  for  t = 1,2  satisfies  these  conditions,  then  x = 
is  the  unique  minimizer  of  (3.1). 


Xi 

x2 


For  given  U\, 

II 

u 

V\(UiU  Li),  and  U2,  L2 

C w,  f2 

= w\(u2ul2), 

we  define 

= 1 if  i £ Ut, 

xu  = 0 

if  i £ Lt  for  t = 

1,2, 

(3,7) 

yu  = 0, 

i'n  = 0 

if  * £ Ft  for  t = 

1,2, 

(3.8) 

, (M 
— 

-™  + Ei£fi!/,)for(  = 

Kd 

1,2, 

(3.9) 

= 

yt,  — xt 

if  i £ Ft  for  t - 

: 1,2, 

(3.10) 

and 

yu  = yt,  - 1 

— Aj  if  i £ Ut, 

Vti  = 

Xt  - yt,  if  i £ L 

t for  t = 

1,2  (3.11) 

From  (3.2),  (3.7)  and  (3.11),  we  have,  for  t = 1,2, 

yt,  = 0 if  * £ Lx  and  uti  = 0 if  i £ Ut. 

We  now  show  that  these  choices  for  Xi,  X2,  yi,  y2,  ^i,  ^2,  Ai  and  A2  satisfy  the  condi- 
tions (3.2),  (3.3)  and  (3.4)  as  following  ; 

/g  9\  . xi,  ~ Uii  + y\,  — vi,  + Ai  = 0 Vi  £ V, 

x2,  — 2/2,  + 7*2,  — ^2,  + A2  = 0 Vi  £ W. 

For  i £ Ut,  since  xt,  ==  1, fiti  = yti  — 1 — Xt  and  ut,  = 0 for  t = 1,2, 

xt,  ~ Ut,  + A h,  ~ vti  + At  = 0 for  t = 1, 2. 

For  i £ Lt,  x<t  = 0,  = Xt  — yt,  and  ylt  = 0 for  f = 1,2.  So, 

Zt,  - yt,  + yt,  ~ + At  = 0 for  t = 1, 2. 

For  i £ Ft,  yt,  = 0,  iq,  = 0 and  xti  = yt,  — At  for  t = 1, 2..  Hence, 


xt,  - yt,  + yt,  - Vt,  + Aj  = 0 for  f = 1, 2. 
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Therefore, 


%ii  — Vu  + AT  ~ ui,  T Ai  — 0 Vi  G V 


and 


x2{  — U2,  T AT  — ^2,  + A2  — 0 Vi  G VT. 


(3.3)  : J2Vi  XU  = m and  E!2  x2,  = m. 

Tl 

5^^!,  = it/ii +x:-i, 

«=1  ieFi 

= It'll  + 5>.-|c.|a, 

i€Fi 

= it/.i  + Et'-.-i^i— 

ieFi 

= m. 


m + E,-6F,yiJ 

Ifil 


EX2- 


|fT|  + ^ tc2> 

1GF2 

|F2 1 + ^ 2/2,  ~ |F2|A2 

iGF2 

|£/jI  + E^-If>I— 

1GF2 

m. 


m + E,-6f2  yzJ 

|f2| 


(3.4)  : yUi, ( 1 — xi,.)  = 0,  — 0 for  all  i G V,  and  fi2i(l  — x2t)  = 0,  v2xx2,  = 0 for 


all  i G IT'. 


Since  xti  = 1 for  i G Ut,  y,t,  = 0 for  i ^ Ut,  xti  = 0 for  i G f ( and  i/ti  = 0 for 
i ^ for  <=1,2, 


<21,(1  — XiJ  = 0 and  uitxii  = 0 Vi  G V 

and 

<22,(1  — x2.)  = 0 and  u2ix2i  =0  Vi  G W. 

Now,  if  we  show  that  the  conditions  (3.5)  and  (3.6),  <ii,/i2  > 0 and  i/x ,i/2  > 0,  and 
0 < Xi , x2  < 1,  are  satisfied,  then  the  choices  for  x,/x,i2  and  A,  (3.7)  — (3.10),  satisfy 
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the  first-order  optimality  conditions  (3.2)  — (3.6),  and  x = ^ **  J is  the  solution  of 
the  quadratic  problem  (3.1).  Suppose  that  y (a)  = x°  + ag,  that  is,  yi(o)  = x°  + agi 
and  y2(a)  = x°  + «g2  where  a > 0 scalar,  0 < Xj,x°  < 1 and  lTx°  = m = 1 r~° 

2 Procedures  to  Determine  Active  Sets  for  Two  Partitioned  Sets 


1 x“. 


To  determine  active  sets  for  a = 0+  and  a — aj , we  need  the  following  two 

procedures,  Starting  Procedure  and  Continuing  Procedure. 

2.1  Starting  Procedure  for  Two  Partitioned  Sets 

Let  B\  = {i  : x°u  = 1},  B°x  = {i  : x°u  = 0},  B\  = {*'  : x°2i  = 1},  5°  = {i  : 


x 


° = 0}  and  Fi0  = V,  F2o  = W i.e.,  Ulo  = Zi0  = 0 and  U2o  = L2o  = 0.  Let 


and 


5i,  = 


Sig Fij  5l, 

\K\ 


92, 


J2ieF2  92, 


7L  = I]  51,-51,,  72,= 

{iCBfWij,  Si, >517}  {ieB$\U2j,  g2,>g2~} 


1^1 

52,  - 52“, 


(3.12) 

(3.13) 


Th  = Y 5i,  -5i,, 
{ieB°\L1},  31,  <317} 


r2,  = ^ 52,  -52,-  (3.14) 

{ieB°\L2j,  32,  <327} 


Define  t7 ij+1  , Li,+j,  U2j+l  and  Z2j+1  . For  each  j G IV,  if  7^  > r1;,  then  put  each 
i € \f7ij  such  that  g1(  > gTJ"  into  ^i,+i,  and  if  7i,  < ri,,  then  put  each  i G B°\Lij+1 
such  that  glt  < ^7  into  L ij+1 . For  each  j G IV,  if  72,  > t2j,  then  put  each  i G fi2  \ U2j 
such  that  g2i  > ~g2 ~ into  U2j+l , and  if  72j  < t2j  , then  put  each  i G B2  \ L2j+1  such  that 
52,  < 527  into  L2j+1.  That  is, 

5, +i  — Uij  u {*  G B\  \ U\j  : g\x  > g^} 

Fl+i  - Y 

Ul,  + l ~ u e ^2  \ U2]  ■ 52,  > 52,} 

6, -+l  = &i 

I'lj+i  = f i,  u {*  £ B\  \ Li,  : gi,  < 517} 

~ L+1  — 

^2,+!  = L2j  U {i  G B“\  L2j  : g2i  < (fiT} 

Lh+i  = ^2, 


if  7j  > Tj, 
otherwise, 
if  7,  > Tj , 
otherwise., 


(3.15) 


if  li  < Tj, 
otherwise, 

if  7j  < Tj , 

otherwise, 


(3.16) 


and 
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Fi,+i  ~ K \ (Uij+i  u Li]+i)  » 

h:+1  = f2o  \ (u 2j+1  u L2]+lj  . 


(3.17) 


Lemma  12  For  all  k > j in  N,  if  7^  > Ti},  then  g\}  > g\k,  and  if  71  < 77  , then 
9if  < 9u-  Similarly,  for  all  k > j in  N,  if  72j  > t2j,  then  gff>gff,  and  if  ^2j  < r2], 


then  g2]  < g2k . 


Proof. 

By  using  the  proof  of  Lemma  1,  we  can  show  that  for  all  k > j in  N,  if 
7ij  > Ti ji  lhen  di 7 > and  if  7i_,  < TiJ?  then  Wf  Si  gif-  Similarly,  we  can  prove 
that  for  all  k > j in  N if  7 2j  > r2j , then  g~2 f>gff,  and  if  7 2j  < r2j , then  gZf  < g^.  □ 
Hence,  by  Lemma  12,  for  t = 1,2, 


9u>9tk  Vi  G Utj,  Vk  > jf, 

(3.18) 

9t,  < 9tf  Vi  e Ltj , Vfc  > j. 

(3.19) 

Let’s  stop  this  process  for  x0  if  7^  = T\}  = 0 and  72j  = t2j  = 0.  Then  since  and 
B\  are  finite,  this  must  be  terminated  in  finite  steps.  Let  s be  the  first  number  such 
that  jls  = Tis  = 0 and  72j  = r2s  = 0,  then  define 

ffio  = Ui,  = Lio  = LU  = Lis+l,  Fi0  = Fls  = Fi0  \ ( Ui,  U Lls), 

and 


U2o  - U2s  - u2s+ 1,  L2o  - L2s  - L2s+l,  F2o  - F2s  - F2o  \ (U2s  U L2s). 


Thus,  by  (3.15),  (3.16),  (3.18)  and  (3.19),  for  t = 1,  2, 

X^t’gFt,  hit,  zL/i£F(0  9ti 


and 


9tk  > 


9tk  < 


\K\  Kl 

ZligFt,  9t,  _ ieF,0  9u  . 


if  keut0  = ut. 


(3.20) 


\F*.\ 


l^ol 


if  k £ Lto  — Lt  . 


(3.21) 
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2.2  Continuing  Procedure  for  Two  Partitioned  Sets 

minl  [llxi  — yi||2  + ||x2  — y2||2] 

subject  to  0 < Xi  < 1,  0 < x2  < 1,  lTXi  = m — T7Xi 
where  l7  = (1, 1,  • • • , 1),  = Xj  + agi  and  y2  = x2  + ag2.  Let 

Uh  = Ihicti)  = {i  : xi = 1}  , U2j  = U2(aj)  = {i  : x2i(atj)  = 1}  , 

Lij  = Li{ctj)  = {i  : xlt(aj)  = 0}  , L2j  = L2(aj)  = {i  : x2i(aj)  = 0}  . 

Lemma  13  If  Ui(ai)  = U\(a2)  and  Lj(qi)  = L\{a2),  then  for  all  «i  < a < a2, 

Ui(a)  = Ui(ai)  and  In  (a)  = Li(«i). 

Similarly,  if  U2(a3)  = U2(a4)  and  L2(a3 ) = L2(a4),  then  for  all  a3  < a < a4, 

U2(a)  = U2{a3)  and  L2(a)  = L2(a4). 


Proof  The  proof  is  the  same  as  the  proof  of  lemma2.  □ 

By  Lemma  13,  we  can  conclude  that  there  are  only  finite  number  of  breaks. 
Define  U\ (aj+1),  U2(otj+ 1),  £i(aJ+1)  and  £2(aj+i)  such  that 

if  gt,(aj+ 1)  = 0,  xti(aj+i ) = 1 then  i e Ut(aJ+l)  for  t = 1,2,  (3.22) 

if  vti(atj+1)  = 0,  xt,(aj+i)  — 0 then  i € £<(aj+1)  for  t = 1, 2.  (3.23) 

Let 


Ft  = Ftj  U Ut(aj+1)  U £t(aj+1),  f/t°  = 0,  L°t  = 0 for  < = 1,2. 

(3.24) 

For  0 < / and  t — 1,2,  define 

37  _ S.'€F‘i  9t, 

9 |F(,|  ’ 

(3.25) 

7*i  = 

M 

1 

(3.26) 

i^t(aj+i)\u‘t,gti>glt 
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and 


Tu=  _(sii-9u). 

ieCt(aJ+i)\Llt,gt,<g[ 


(3.27) 


Also,  define 


u',*'  = u‘ 

L\+l  = L[  U ji  € £((qj+i)  \ L\  : gti  < g\ | 


U\+l  = Ult  U |z  € Wi(aJ+i)  \ U[  : gt,  > g\  J if  jtl  > ri( 

otherwise  , 

if  It,  < rtl 
otherwise  , 


1+1  _ T i 


L\ 


and 


Ft*1  =F?\  ((/,'+'  Ui|«). 


(3.28) 

(3.29) 

(3.30) 


Let  us  stop  this  process  if  7*,  = 77  = 0,f  = 1,2.  Apply  Lemmal2  with  B} 
Ut(aj+i),B°  = £t{aJ+i),Fto  = F°  and  7<J+1  = 7t,,rt+1  = Tt„t  = 1,2.  Then  si 


since 


Ut{ctj+ 1)  and  Ct(aj+i)  for  t = 1,2  are  finite,  this  must  be  stopped  in  finite  steps.  Let 
s be  the  first  number  such  that  jtt  = = 0,1  = 1,2,  then  define  Utj+l  and  Ltj+l 

with  Ftj+l  = (Ut}+1  U Lt]+1)c  for  t = 1,2  as  follows  : 


u,,»  = (utl\u,(aJ+1))uu; 


and 


Lt}+ 1 — {Ltj  \Ct{&j+ 1))  U Lst, 
where  as  similar  to  (3.20)  and  (3.21), 

9u 


(3.31) 


(3.32) 


9tk  > 


if  k € Uts,  t = 1,2, 


0+1 1 


and 


YlieF,  ,,  9t, 

gtk  < ■ ,FJ+1.  iffceL*,  t = 1,2. 

Tb+il 


(3.33) 


(3.34) 


Now  we  define  functions  x(Fj,aj),/j,(Fj,ctj),  \(Fj,ctj),  with  0 = <a0  < Qi  < 
q2  • • •,  where  (Ftj+1,Oj+i)  are  obtained  from  ( Ft],otj),t  = 1,2  in  the  following  way; 
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and 


For  t = 1,2, 


At(a) 


yti(a)) 

|Ff  | 1 5 if  Qj  < Q < Oij  + 1 

\Ut  l-m  iteF,  x ot,  “EigF, . fltj 

+ — 1^—  + i- 


1^1 


1^1 


Me*)  = y<,(a)  - A(F(j,  a),  if  i G Ftj  and  ctj  < a < aJ+l 
= xt0 . + a ( 0t,  - 


E,gFtj  StA  |C/t)|-m  E.£Ft)^o, 

v F‘jl  y _ 1^1  “ F.,1  ’ 

Me*)  = yt,(Q)  - 1 ~ A {Ftj,a),  if  i G Utj  and  cij  < a < aj+l 

( E.ep,  9t,\  \ut\-m  E,en,xt o, 

- z<0,  1 + |Ftj|  J |Ftj|  ]H~T  ’ 

Ft,  (a)  = A(Ftj,a)  - y<,(a),  if  i G Ttj  and  a:  < a < aJ+l 

\Ut.\-m  EigFj.^io,  /EigF;  St; 

; + — IFH — + a 


Ifi, 


Fh 


Ftjl 


i 


9t,  ~ xt0 


aj+i  — min  < 


sup{a  : yi,(a)  > 0 Vi  G U1},fiu(ap  ± 0}, 
sup{a  : uu(a)  > 0 Vi  G Llj,ult(af)  ± 0}, 
sup  {a  : 0 < xu(Fl3,a)  < 1 Vi  G Flj,xli(aJf)  / 0, 1}, 
sup{a  : y 2, (a)  > 0 Vi  G U2] , y2, (<*+ ) + 0}, 
sup{a  : J/2i(a)  > 0 Vi  G L2> , F2j  (<*+)  ± 0}, 
sup{a  : 0 < x2,(F2j,a)  < 1 Vz  G F2j , x2, (a+ ) ^ 0, 1}  / 


By  (3.2),  (3.36),  (3.37),  (3.38)  and  (3.39),  for  t = 1,2,  we  have 


(3.35) 

(3.36) 

(3.37) 

(3.38) 


(3.39) 


and 


/Me*)  = 0 

for  i f Ut] , ctj  < a < ai+l , 

(3.40) 

Ft,  (a)  = 0 

for  i ^ < a < aJ+1, 

(3.41) 

xu{a)  = 1 

for  i G Utj,cij  < a < aJ+i 

(3.42) 

xtfa)  = 0 

for  i G Ff  , < a < aj+1. 

(3.43) 

i and  a,  xtl( 

a),  At(a),yt,(c*)  and  utt(a) 

are  continuous  piece- 

wise  linear  functions  of  a,  and  0 < xt,(e*)  < 1 for  t = 1,2,  that  is  0 < xj  < 1 and 

0 < x2  < 1. 


This  theorem  is  followed  by  theorem  3. 
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Hence,  the  condition  (3.6)  is  satisfied.  Now,  if  we  hold  the  condition  (3.5), 
fit  > 0,  ut  > 0 for  t = 1, 2,  then  x(a)  = ^ is  the  solution  of  (3.1).  Let 

Ftj  = (B/  U B°)c  where  Bj  = {i  : xto.  = 1},  Bt°  = {i  : xto  = 0},t  = 1,2.  Then  by 
the  same  argument  as  before  in  previous  chapter,  we  can  show  the  condition  (3.5)  is 

Xi(q) 

x2(a) 

We  have  the  same  argument  for  extreme  point  of  convex  set  and  stopping 
criteria  for  a given  search  direction  as  before. 


^ is  the  unique  minimizer  of  (3.1). 


hold.  Therefore  x(a) 


CHAPTER  4 
ALGORITHM 

We  will  implement  algorithm  to  obtain  a local  minimizer  of  the  quadratic 
programming  problem 

min/(x),  /(x)  = (1  - x)tAx 
subject  to  0 < x < 1,  l/x  = m 

by  using  gradient  projection  method  with  the  step-size  amin  such  that 

/(x(amin))  = min/(x(a)) 

(>>0 

where  x(a)  = ProjK[xk  - aV/jx*)]. 

Let  U , L be  upper  and  lower  active  sets  of  indices  at  current  iteration,  respec- 
tively. F = (U  U L)c . That  is,  F : free  set,  inactive  set. 

Let  gk  : — V/(xfc)  = -(Al  -2Axfc  ),g  = and  z where 

c = 0 at  the  starting  point.  Let  I57?,  g sf  and  XoSF  are  vectors  which  have  length  of 
n with  1,  gi  and  x0,  respectively  for  all  components  for  i G F and  0 for  others.  1 su 
and  1 sl  are  vectors  which  have  length  of  n with  1 for  all  components  for  i G U,  i € L 
respectively  and  0 for  others. 

4f{x(a))  _ dxdf(x(a)) 
da  da  dx 

(g f — ^1f)T[Af1  - 2Afx(q)] 

(gF  — f)1  [AffIf  + Apu^u  + AflIl  — 2A  ffXf(o)  — 2Aff1f] 

(g  f — 9^f)T[Aff1f  — 2A  ffX-f(cx)  — AffIf  + AffIf] 
g fAf  [(1  + 2 ag  + 2c)1sf  — 2(agsF  + xoSF)  + (Isl  — Isf)] 

— IfA f5  [(1  + 2 ag  + 2c)1sf  — 2(agsF  + Xq sf)  -f  (Isf  — Isf)]  • 


/'(x(q))  = 


35 


36 


Figure  4.1:  Outline  of  program 
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Figure  4.2:  Outline  of  partition  of  the  original  graph 
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/"(x(a)) 


dx  d2f(x(a))  / dx\T 
da  dx2  \da/ 

(g f ~ ~9~^-f)T{  2A/rf,](gf  - glF) 
-2{EfA-FF  - g^^FF){gF  - #1f)- 


1 Initial  Point 


We  can  get  an  initial  point  in  the  following  three  ways  : 

(i)  xc,  the  centroid  of  the  convex  set,  K = {0  < x < 1,  l7x  = m) 

(ii)  The  point  which  has  the  best  function  value  of  the  ball  centered  at  xc  in  K. 

(iii)  xc  + y,  the  projection  onto  K of  the  point  whose  function  value  is  the  best  of 
the  ball  which  contains  K. 

1.1  The  Solution  of  the  Ball  Centered  at  x,~  in  I\ 


Let  xc  + y be  the  solution  of 

min  (xc  + y)TA(l  - xc  - y) 
subject  to  ||y||  < r,  lTy  = 0 


where  xc  is  the  centroid  of  K,  —1  and 

L ’ n 


(4.1) 


r 


(II  XC 

min 


— the  closest  point  on  the  boundary  of  K fromxc||) 

‘V2-,  V5(l--)1. 

In  V n / 


Then  the  radius,  r is  the  minimum  distance  from  centroid  to  boundary  of  I\  and  so, 
0 < (xc  + y),-  < 1 for  all  i.  Since 


(xc  + y)TA(l  - xc  - y)  - xf  A(1  - xc)  = -yrAy  + yTA(l  - 2xc), 


the  solutions  for  (4.1)  is  equal  to  the  solution  of  the  following  : 


min  — yrAy  + yrA(l  - 2xc) 
subject  to  ||y ||  < r,  l7y  = 0 


(4.2) 
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where  1 - 2xc  = (1  - ^f)l. 

Let  f i , f 2 , • • • ,fn  be  orthonormal  eigenvectors  corresponding  to  the  eigenvalues, 
Ai,  A2,  • • • , A„  of  PAP  where  P = I — All7.  Take  fi  = ^1,  Ai  = 0 and  y = ^”=2  c jfi- 
Then 


PAPf,  = A,ft 


and  since  l7y  = 0,  Py  = y.  So, 


Also, 


Hence, 


and 


yTAy  = yTPAPy 


yT]TPAP(ctf,) 

i= 2 

' n \T  n 

^ / c,fj'  J ^ ' CjAjfj' 

v,  1=2 

E*. 


8 = 2 


8 = 2 


yrA(l  - 2xc) 


|>)  A(1-v)1 


= Ec'(‘ 


1=2 


v)  ^A1 


= min  — yTPAPy  + yTPA(l  - 2xc) 
subject  to  ||y||  < r,  lTy  = 0 


(4.3)  = 


. min  £"=2  (— c? At  + 2dtCi) 


subject  to  Y%=  2 ci  < ?’2,  1T  ZIL2  ci£  = 0 


(4.3) 


(4.4) 


where  d{  — | (l  — ^-)f7PAl.  If  Py  = y,  then  l7y  = 0.  Hence  the  solution  for 
(4.2)  is  equal  to  the  solution  of  the  following  problem  : 

min  f EIU  (-ci2Ai  + 2 did) 

subject  to  A £”=2^<t-  ^ 

Lemma  15  If  PAP  has  a positive  eigenvalue,  then  ]T)”_2  c2  = r2  at  the  minimum  of 
(4.5). 
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Let  Xi  = 1 ,xj  = —1  and  Xk  = 0 for  all  k / i,j.  Then  Px  = x. 


xtPAPx  = (Px)T  A (Px)  = xtAx. 


If  a,ij  = 0 for  some  i,j,  then  a,-,'  + djj  = 2 > 0.  Thus  PAP  has  a positive  eigenvalue. 
Since  PAP  has  a positive  eigenvalue,  by  the  lemma  15, 

min  \ EL 2 (~c.2^  + 2 diCi) 

~ subject  to  \ £”=2  c]  = y. 

Let  us  see  the  form  of  a solution  to  (4.6)  by  first  and  second  order  optimality  condi- 
tions. At  minimum,  let  y be  a multiplier  for  constraint,  then 


-CjAt  T (l i + 1 1 c i — 0 
Ci 

E 


c = — — 

Ct  m-A; 

dl 


=2  (/x-A.)2 


= r2. 


Since  \ EL 2 (— + 2d;Ci)  and  | EL2  cj2  are  in  C2>  we  can  use  the  second 
order  optimality  condition.  Hence  if  c is  a local  minimizer  of  (4.6),  then  for  all  y 6 Rn 
such  that  EL2  = 0? 


But  (V2(|  EL2  (~ cl^i  + 2diC,))  + yuV2(|  EL2  A2))  is  a matrix  whose  diagonal  is 
(0,/r  — A2,  y — A3,  • • • , y — An)  and  all  the  other  entries  are  zeroes.  Hence,  (4.7)  is 
equivalent  to 

n 

- K)Vi  > 0.  (4.8) 

i- 2 

Thus,  by  the  second  order  optimality  conditions,  we  know  that  the  solution 
to  (4.6)  can  be  expressed  in  the  following  way:  Let  y be  the  largest  number  such 


that 


£ 


n 

§ (>■  - w 


= r 


(4.3)  = 


min  — yTPAPy  + y'  PA  (1  - 1 

subject  to  yTy  = r2,  lry  = 0. 


(4.9) 

(4.10) 
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Hence,  by  using  the  projection  onto  1,  P = I — -1 17  and  the  first  and  second  order 
optimality  conditions,  we  know  that  the  minimum  occurs  at  the  boundary  and  the 
form  of  the  solution  of  (4.2)  is  (4.9). 

Now,  we  will  find  a solution  in  Rn  of  (4.2).  We  can  find  a unit  vector  Vi 

which  is  orthogonal  to  1 from  a random  vector  on  the  unit  sphere  in  R"-1  by  using 

Householder  transformation.  Let  v1?  • • • , v*,  be  orthonormal  vectors  for  which  1 1 v,-  = 

0,  i = 1, 2,  • • ■ , k and  y = Vz  where  V is  the  matrix  with  columns  of  V{  s.  Then  since 

zl  z — z/V7Vz  = yTy  and  l7y  = 0,  we  have 

,.  _ min  (— zTVTPAPVz  + zTVTf ) 

~ subject  to  zTz  = r2. 

where  f = PA  (l  — ^p)  1.  Let  PAP  = S — Ti  where  S is  invertible  and  M = S_1Ti. 

If  Vfc  denotes  the  matrix  whose  columns  are  {vi,  • • • , v*,}  and  H is  the  (k  + 1)  x k 

upper  Hessenberg  matrix  whose  elements  are  given  by  the  Arnoldi  process,  then 

MVt  = Vi+1H. 


Let  S = — iul,  then  since  Tj  = S — PAP, 


M = 1 + -PAP. 

w 

Let  w > p(PAP).  Then  w > — (smallest  eigenvalue  of  PAP)  and  so  all  eigenvalues 
of  M are  positive.  Actually,  since  our  matrix  M is  symmetric,  H is  the  (k  + 1)  x k 
tridiagonal  matrix  (see  [8]). 

VtPAPV  = Vt(S-T!)V 
= VtS(I-M)V 
= SVj(V,-Vfc+1H) 


= S(I-T2) 


= — ru(I  — T2) 


where  T2  is  tridiagonal  since  H is  tridiagonal. 
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1.  Generate  a random  vector  x on  the  unit  sphere  in  Rn_1. 

2.  Generate  a unit  vector  Vi  with  l1  Vi  = 0 from  x. 

3.  Use  Arnoldi  algorithm  to  generate  V and  T2  with  above  Vj . 

4.  Use  the  following  algorithms(Newton  And  Bisection)  to  solve 

min  - (ztT3z)  + t7  z,  t = VTf 
— subject  to  zTz  = r2 


(4.12) 


where  T3  is  — tu(I  — T2)  above. 

5.  y = Vz  (This  will  be  the  solution  of  (4.2)). 

We  will  use  Arnoldi  process  to  choose  V to  be  a matrix  whose  columns  are 
orthonormal  and  V;PAPV  is  tridiagonal.  If  the  starting  vector  for  the  Arnoldi 
process  has  components  summing  to  zero,  and  if  we  use 


M = 1+  -PAP, 

w 


then  all  the  succeeding  vectors  satisfy  this  condition  (summing  the  components  is 
zero).  At  first,  generate  a random  vector  x on  the  unit  sphere  in  Rn-1  by  using 
Gaussian  distribution  with  mean  = 0 (see  [8]).  Let 

x = (xi  • • • xn_i)T  and  x = 

Now  choose  H = I — 2wwr  such  that 


HI  = 


/ 0 \ 


0 

V1/ 


Since  H is  symmetric,  the  last  row  and  column  of  H are  ^1T  and  -^1.  ||Hx||  = 1 
since  x is  a unit  vector.  H rotates  1 to  the  n-th  axis,  and  when  a vector  on  the 
n-th  axis  is  multiplied  by  H,  it  is  rotated  to  1.  By  orthogonality,  a vector  which 
is  orthogonal  to  n-th  coordinate  axis  is  mapped  into  a vector  orthogonal  to  1.  Let 
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a = y/n( 0 • • • 0 1)T.  Then 


1 — a 

w = n rr 

||1  - a|| 2 

= 1 —(1 

V(n  - !)  + (1  - V™)2 

Hence, 


and 


H 


= I 


I — 2ww7 

1 


n — \/n 


(1  •••  1 1 - V^)T(1  •••  1 1 - vH 


Hx  = H 


/ 

( 1 ^ 

\ 

( X\  \ 

1 — V 

n — sjn 

1 

(1  * 

• 1 1 — y/n) 

Xn—\ 

1 — \fn 


x 
x — 


x7l 


n — wn 


1 

1 - \fn 


V" 


Let  v i 


XT1 

\/n 

xrl  ^ 

n— v/n 
xTl 
\fn 


. Then  V\  is  a unit  vector  which  is  orthogonal  to  1. 


Algorithm  (Arnoldi  for  symmetric  version) 


Vi  is  given 
for  j = 1 : k — 1 
s <—  Mvj 
for  * = j - 1 : j 

hij  vfs  (=vfMvj) 

S i S hijVi 

end 

hj+i,j  «—  ||s|| 
if  3i+i,j  = o, 
stop 

end 

vj+i  t-  s/hj+u 


end 
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end  Algorithm 

By  Arnoldi  process,  we  generate  V and  T2.  Then  we  can  get  T3  and  t in 
(4.12).  Now  we  are  ready  to  solve  (4.12). 

Let  us  consider  the  problem  of  the  form 


mm 


{f>(w)  : ||ty||  < r} 


1 


if{w)  --  -wT Bw  + g Tw, 


(4.13) 


where  r is  a positive  parameter,  ||  • ||  is  the  Euclidean  norm  in  Rn,  and  with  g € Rn, 
and  B £ R"Xn  a symmetric  matrix. 


Lemma  16  If  p is  a solution  to  (4.13)  then  p is  a solution  to  an  equation  of  the 
form, 

(B  + A/)p  = — g,  (4.14) 

with  B + XI  positive  semidefinite,  A > 0,  and  A (r  — ||p||)  = 0. 

Lemma  17  Let  A £ R,  p £ En  satisfy  (4.14)  with  B + XI  positive  semidefinite. 

1.  If  A = 0 and  ||p||  < r then  p solves  (4.13). 

2.  p solves  4>{p)  = min{t/>(rc)  : ||ro||  = ||p||}. 

3-  If  A > 0 and  ||p||  = r then  p solves  (4.13). 

If  B + XI  is  positive  definite,  then  p is  the  only  solution  to  (4.13). 

See  [19]  for  the  proof  of  Lemma  16  and  Lemma  17.  Note  that  these  lemmas 
provide  necessary  and  sufficient  conditions  for  a point  p £ R"  to  be  a solution  to 
(4.13)  and  that  there  is  no  gap  between  the  necessary  and  sufficient  conditions. 

(4.12)  = min_!zT(-T)z  + t7z 

v ' ' subject  to  zTz  = r2. 


(4.15) 
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where  T = 2T3  = — 2io(I  — T2).  Let  p be  the  solution  of  (4.15),  then  p7p  = r2. 
Hence,  if  we  have  — T + AI  positive  definite  with  A > 0 then  p = — ( — T + AI)-1t  is 
the  solution  to  (4.12).  But  if  t = 0,  then  it  is  not  true.  So,  we  can  use  this  only  if 
t ^ 0.  We  will  discuss  t = 0 case  later. 

Assume  t ^ 0.  Let  Amax  be  the  largest  eigenvalue  of  T.  If  A > Amax,  then 
— T + AI  is  positive  definite.  We  can  evaluate  the  upper  bound  of  the  eigenvalues  of 
T by  Gershgorin’s  theorem.  Let 

Pa  = -(-T  + al)_1t. 


We  know  that  the  function  ||pa||2  is  a rational  function  in  a with  second  order  poles 
on  a subset  of  the  positives  of  the  eigenvalues  of  the  symmetric  matrix  T as  (4.9). 
The  rational  structure  of  ||pa||2  may  be  exploited  by  applying  Newton’s  method  to 
the  zero  finding  problem 

0(a)  = |[pa||  — r = 0.  (4.16) 


If  0(a)  is  positive,  Newton’s  method  is  very  efficient  when  applied  to  (4.16)  since  0 is 
monotone  decreasing  on  (Amax,oo).  Moreover,  the  computation  of  the  Cholesky  fac- 
torization of  — T + al  makes  it  possible  to  compute  the  necessary  derivative  whenever 
a E (Amax,oo).  Let  RTR  is  the  Cholesky  factorization  of  — T + AI  with  R E RnX" 
upper  triangular  and  p be  the  solution  of  RTRp  = — t. 


0'(a) 


T ' 

P P 


(-(-T  + al)-H)T  (— T + al)_1l(— T + aI)"H 


(R-1R-Tt)T  R-1R"TR-1R-Tt 


—r. — r7tTR_1R_TR_1R-rR_1R_Tt 

IpII 

-pTR-1R_Tp 
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where  Rrq  = p.  Therefore, 


m ...  it p 11  ( 11  p 11  ~ r) 

^'(A)  |[q||2 

Hence,  without  computing  the  eigensystem  of  T,  the  following  algorithm  (Algorithm 
3.2  in  [19])  updates  A by  Newton’s  method  applied  to  (4.16). 

Algorithm  (Newton)  Let  A > 0 with  — T + AI  positive  definite  and  r > 0 be 
given. 


1.  Factor  — T + AI  = RTR  ; 


2.  Solve  R7Rp  = — t ; 

3.  Solve  R;  q = p ; 

4.  Let  A = A + ; 

end  Algorithm 

We  know  that  in  general,  Newton’s  method  is  not  guaranteed  to  converge. 
If  q satisfies  that  </>(a)</>  (a)  > 0 then  the  iterations  converge  monotonically.  Since 
cf>  (a)  is  positive  in  (Amax,oo),  in  fact  4>(a)  is  monotone  decreasing  in  (Amax,oo),  if 
</>(a)  is  positive,  then  Newton’s  method  is  guaranteed  to  converge  in  (Amax,oo).  If 
cf>(a)  is  negative,  then  next  Newton  iterate  A may  less  than  Amax  and  converge  to  less 
than  Amax.  To  make  — T + AI  positive  definite,  A must  stay  in  (Amax,oo).  Hence,  we 
need  to  find  A > Amax  such  that  </>(A)  > 0.  By  using  Gershgorin’s  theorem,  we  can 
find  the  upper  bound  of  eigenvalues  of  T.  Let  the  initial  A be  the  upper  bound  of 
eigenvalue  of  T,  then  A > Amax.  Repeat  the  following  algorithm  until  we  get  A > Amax 
and  </>( A)  > 0. 

Algorithm  (Bisection) 

A = \new  = upper  bound  of  the  eigenvalues  of  T 
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evaluate  cf>( A) 

do  while  A < Amax  or  <f)( A)  < 0 
call  Algorithm  (Newton)  to  find  Ar 


do  while  A new  A Amax 


\ — Anetl,-f  A 

''new  — 2 


end 

A = Ar 


^ Amax) 


evaluate  </>(A) 
end 


end  Algorithm 

Now  let  us  see  how  to  check  either  A > Amax  or  A < Amax.  In  general,  let  fk 
denote  the  determinant  of  the  fcth  leading  submatrix  of  a matrix  C.  Then 


fk  — C-kkfk—l  Pkfk- 2i  (4.17) 

where  pk  = Ckk-iCk-ik  is  the  product  between  elements  on  opposite  side  of  the  diag- 
onal and  the  initial  values  for  the  fk  are  /0  = 1 and  /_i  = 0. 

The  recurrence  (4.17)  also  provides  a formula  for  the  characteristic  polynomial 
det  (C  — AI).  If  C is  a tridiagonal  matrix,  C — AI  is  the  tridiagonal  matrix  obtained  by 
subtracting  A from  each  diagonal  element  of  C.  Letting  /*.( A)  denote  the  determinant 
of  the  kth  leading  submatrix  for  C — AI,  fk( A)  satisfies  recurrence  (4.17)  except  that 
Ckk  is  replaced  by  Ckk  — A : 


fk{ A)  = ( Ckk  ~ A)/fe_i(A)  - pkfk-2{ A),  (4.18) 

where  /0(A)  = 1 and  /-i(A)  = 0. 

We  can  check  either  A > Amax  or  A < Amax  by  the  Sturm  sequence  property 
: For  a n x n tridiagonal  matrix  with  each  pk  nonnegative,  the  number  of  eigen- 
values smaller  than  A is  equal  to  the  number  of  changes  in  sign  for  the  sequence 
/o(A),  /i(A),  • ■ ■ , /„( A).  We  know  that  T is  a symmetric  tridiagonal  matrix  and  so, 
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the  pk  are  always  nonnegative  : pk  = tkk-itk-ik  = {hk- i)2-  Hence,  the  number  of 
eigenvalues  smaller  than  A is  equal  to  the  number  of  sign  changes.  Thus,  if  there  is 
no  consequtive  positive  or  negative  signs,  then  A is  greater  than  Amax. 

By  above  two  algorithms  (Newton  and  Bisection),  we  can  find  p,  the  solution 
of  (4.12)  for  t 7^  0. 

Let  y = Vp,  then  this  y is  the  solution  of  (4.3)  and  of  (4.1).  Hence,  we  will 
take  xc  + y as  the  starting  point. 

Now,  suppose  t = 0.  Then 


(4.1-2) 


min  — ztTz 
subject  to  zTz  = r2. 


(4.19) 


The  solution  of  (4.19),  p,  is  an  eigenvector  corresponding  to  the  largest  eigenvalue  of 
T and  Vp  is  an  eigenvector  of  the  largest  eigenvalue  of  PAP.  To  find  this  eigenvector, 
we  can  use  shifted  power  method.  For  this  case,  we  want  to  make  a biggest  ball  which 
may  not  in  the  convex  set  K,  but  the  solution  of  (4.3)  is  in  K.  So,  we  multiple  some 
scalar  to  the  eigenvector  of  PAP  so  that  xc  + y = xc  + r ■ Vp  is  on  the  boundary  of 
K.  Let 


rt  = 


1-XC, 


Vp, 

-Xc: 


Vp, 


ifVPi>0 
if  V Pi  < 0 


and 


r = min(r,). 
i 

Then  this  r will  be  the  radius  of  the  ball  which  may  not  in  the  convex  set  K,  but 
the  solution  of  (4.3)  is  in  K.  Hence,  we  will  take  xc  + y = xc  -f  r • Vp  as  the  starting 
point. 

1.2  The  Solution  of  the  Ball  Centered  at  xr  Containing  K 


= (||xc  — an  extreme  point  of  K 


= \ m 


n 


r 
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where  xc  is  a centroid  of  K,  —1. 

c 7 n 

By  the  same  way,  we  can  get  the  exact  solution  of  the  ball  centered  at  xc 
containing  K. 

1.3  The  Projection  on  I\  of  the  Solution  of  the  Ball  Containing  I\ 

If  we  get  an  exact  solution  of  the  ball  which  contains  K , we  need  to  evaluate 
the  projection  of  the  exact  solution  of  the  ball  onto  K.  We  will  use  the  optimality 
conditions  to  evaluate  the  projection  of  the  exact  solution  of  the  ball  onto  K. 

mini||x-y||2 

subject  to  0 < x < 1,  lTx  = m \ • ) 

where  y G Rn  is  given.  The  solution  to  the  probelm  (4.20)  is  Proj^iy)-  Since  ||x — y || 2 
is  strongly  convex  function,  there  exists  a unique  minimizer  and  the  following  first- 
order  optimality  conditions  hold;  there  exist  A £ R1  and  y,v  £ Rn  such  that 


x-y  + /r-j/  + Al  =0, 

(4.21) 

lTx  = m, 

(4.22) 

yT(l  — x)  = 0,  z/Tx  = 0 

(4,23) 

y > 0,  v > 0 

(4.24) 

0 < x < 1 

(4.25) 

Conversely,  if  x,/r,  u and  A satisfies  these  conditions,  then  x is  the  unique  minimizer 
of  (2.1). 

For  given  U,  L C {1, 2,  • • • , n},  and  F = (U  U L)c , we  define 


if  i € U, 

X{  = 0 

if  i € L 

(4.26) 

yt  - 0,  Vi 

= 0 

if  i G F 

(4.27) 

A - ^ 

-m  + EJGF2/i) 

(4.28) 

1^1 

Xi  -- 

Vi  - A 

if  i G F 

(4.29) 

X{  — 1 
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Hi  = Ui  — 1 — A if  z £ U,  — A — y,  if  z £ L (4.30) 

Then  we  can  show  that  the  conditions  (4.21)  — (4.23)  are  satisfied  as  before.  Hence, 
we  only  need  to  show  (4.24)  and  (4.25). 

Let  y be  the  exact  solution  of  the  ball.  Let  £/0  = To  = 0 and  F0  = ( UoUL0)c  = 

{1,2, 

Define 

A, = ,4.31) 

Hi  I 

and  for  i £ Fj, 


Aj  = {i  £ Fj  : ax  — yi  — 1 — A j > 0} 
Bj  - {i  £ Fj  : bi  = Aj  - y,  > 0} 


(4.32) 

(4.33) 


Let  t j = Y^ieAj  a»  anci  ri  = YlieBj  ^i-  Define  Uj+i  and  LJ+i  as  follow:  For 
each  j,  if  7 j > Tj,  then  put  each  i £ Aj  into  Uj+ 1,  and  if  7 j < Tj , then  put  each  i £ Bj 
into  Lj+ 1.  That  is, 

TT  — TT.  1 1 A . \ - 

(4.34) 

(4.35) 

and 

Bj+i  = Fo\  {Uj+i  U Lj+i).  (4.36) 

If  we  stop  this  iteration  when  7 s = rs  = 0,  i.e.,  0 < 27  = ys-  — As  < l,Vz  € Fs, 
then  by  the  following  lemma,  we  can  see  that 


Uj+ 1 

= Uj  u Hj 

if  7 j > Tj 

tf;+i 

= u, 

otherwise  , 

A?+x 

= Lj  U Bj 

if  7 j < Tj 

T,+i 

= Lj 

otherwise  , 

Hi  yi  1 A 5 > 0 Vz  £ l' s 1 

Vi  — Xs  yi  > 0 Vz  £ Ls 


and 


0 < x < 1. 
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Lemma  18  If  jj  > Tj,  then  X j > A k for  all  k > j,  and  if  ^j  < Tj,  then  A j < A k for 
all  k > j . 


Proof.  First,  let  us  consider  the  case  7 j > Tj.  Let  / be  the  first  index  greater  than 
equal  to  j for  which  7 ;+1  < r;+1.  Hence,  7 ,■  > rt-  for  all  j < i < l.  In  (4.34),  we 
take  out  from  Ft  for  each  i.  Thus,  Xj  > • • • > A;  > A;+i  and  so,  if  j < k < l + 1 
then  Xj  > Xk.  Let  k be  the  first  index  greater  than  l for  which  7^  > rk.  Hence, 
7 i < T{  for  all  l + 1 < i < k.  In  (4.35),  we  remove  Bi  from  Ft  for  each  i.  Thus, 
A/+i  < • • • < Afc_x  < Afc.  We  will  show  that  Xj  > Xk.  Since  Xj  > • • • > A/,  it  is  enough 
to  show  that  A/  > A*.  Note  that  Xj  = A*  holds  if  j = / and  7 j = Tj  and  A{  = Bi  = 0 
for  j < i < k.  By  way  of  contradiction,  suppose  A*  > A/.  Without  loss  of  generality, 
we  can  assume  Xk  > X / > Afc_i,  be., 


A;+i  < • • • < A^_i  < A;  < Afc. 


Then 


A* 


\Uk\-m  + JfieFk  yi 

\Fk\ 


(4.37) 


(4.38) 
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M - m + E.sf,  ^ - 2E,c.4,  (y.  - 1)  - E‘2+i  E,£b,  ;/ 

Iftl 

I Wi  - E,e,7.  - 1)  - EjE+i  E ,€B, ». 

Ifti 


Hence, 


x x (l-^l  YlieA,(y*  1)  Yiq=i+i  YhieBq  Vi 

A‘“A,= (fti ' 


k — 1 


Since  Xk  > A/, 

di^i  — |fii)A,  — — i)  — E E»->° 

«'eAi  <J=/+1  iefi. 

On  the  other  hand,  since  7 / > r/,  we  have 


^(j/i  — 1 — A,)  > J^(A/  - yj). 

tE-Aj  tE-Bj 

Hence  by  (4.42),  we  have 

fc— 1 

m - m\,  - Y(«‘  - u - E E »•  > 0 

ieAi  q=i+ 1 ieBq 

k—  1 

=*  - ^2(Vi  - 1 - A,)  - - A,)  > 0 

ieAi  q=l+ 1 t€£9 

/c—  1 

=>•  ^2  ~ y •■)  ~ - 1 - A;)  > 0 

g=/+l  iE.Bg  iEA\ 

k—  1 

- EE  ( Aqr  2h)  ^(A/  2/* ) > 0 

g=/+l  i£Bj 

Since  for  l + 1 < q < k — 1,  F9  is  monotone  decreasing  and  A?  < Ay , we  have 


Bq  C Bi 


and  so, 


k—1 


k- 1 


EE  (A,  - y,-)  < EE  (a  1 - yi)  < X(A* _ yi)- 

q=l~\~  1 iEBq  q=l~ t"l  iEBq  iEB\ 

This  contradicts  (4.44).  Therefore,  A/  > A*,  if  k satisfies  7,  < r,  for  all  / + 1 < 

Hence  Aj  > A 


(4.39) 

(4.40) 

(4.41) 

(4.42) 

(4.43) 


(4.44) 

(4.45) 
i < k. 
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Now  consider  a general  k > j.  We  will  use  mathematical  induction  to  show 
that  Xj  > Xk  for  all  k > j.  Let  fc,-  be  the  z-th  number  after  j such  that  7^,-1  < 1 

and  7 ki  > 77, . We  have  seen  that  A j > A*,  for  j < k < k\.  Suppose  it  is  true  for 

j < k < k{.  Then  Xj  > A^.  Let  ki  < k < kl+i,  then  by  the  above  argument,  we  have 
> Xk.  Hence,  Xj  > Xk  for  all  k > j.  Similarly,  we  can  show  that  if  7 ,•  < Tj , then 
Aj  < Afc  for  all  k > j.  □ 

Hence,  by  Lemma  18, 

Hi  = Hi  - 1 - Afc  > 0 Vi  € Uj,  Vk  > j (4.46) 

fi  = Xk  - yi  > 0 Vi  e Lj,  \/k  > j.  (4-47) 

Let  us  stop  this  process  if  7 j = Tj  = 0.  Then  since  F0  is  finite,  this  must  be 
terminated  in  finite  steps.  Let  s is  the  first  number  such  that  7 s = ts  = 0,  then 
define  U = Us  = Us+ 1,  L = Ls  = Ls+ 1,  F — Fs  = F0  \ (Us  U Ls)  and  A = As.  Hence, 
for  all  i, 

Hi  > 0 and  z/j  > 0 

and 

0 < xf  < 1. 

Thus,  x in  /V  such  that 

xt  = 1 if  i G U,  Xi  = 0 if  i £ L and  Xj  = — A if  i E F 

is  the  projection  of  y onto  K. 

2 Starting  Procedure 

(1)  First  of  all,  put  all  indices  in  F and  evaluate  the  new  direction  vector  g 
with  x0.  Then  U — 0 and  F = 0. 

(2)  Evaluate  g = — V/(x0)  = — (Al  — 2Ax0). 

(3)  By  using  Index  Decision  procedure,  determine  U,  L and  F at  a — 
with  U = {i  : x0,  = 1},  C = {i  : x0i  = 0}. 
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(4)  Evaluate  /((x(0))  and  /”(x( 0)) 

/(x(a))  |a=o  = g?AF[(l  + 2c)lF  - 2x0f  + (1L  - 1[/)] 

— 1fAf^[(1  + 2c)  1 p — 2x0f  + (1  l — It/)] 

/ (X(Q))  |o=0=  -2(gpA FF  — g^FAFF)(gF  - </1f) 

(5)  If  gt  >gVi£  Do,  gi  < <?  Vi  € L0  and  gt  =gVi£  F0 , then  go  to  error  evaluation, 

else  go  to  continuing  procedure. 

3 Continuing  Procedure 

(1).  By  definition  of  aJ+1,  find  new  alpha(aj+i)  which  is  larger  than  old 
alpha(ofj). 

( sup{a  : 0 < Xi(Fj,a ) < 1 Vi  6 Fj,Xi(Fj,aJ)  / 0 or  1}, 
a j+i  = min  < sup{a  : /r,(a)  > 0 Vi  € Uj,fii(Fj,af)  ^ 0}, 

( supja  : 17(a)  > 0 Vi  G Lj,Vi(Fj,a+)  ± 0} 

(2).  By  using  the  following  procedure,  Index  Decision,  determine  new  U,  L and  F 

at  a — ctj+\. 

3.1  Index  Decision 

Define  U and  £ such  that 

if  m{a.j+ 1)  = 0,  x,(aJ+1)  = 1 then  i € U, 
if  Ui(aj+ 1)  = 0,  Xi(ctj+ 1)  = 0 then  i € £. 

To  determine  new  U,  L and  F at  a = aJ+1  repeat  the  following  until  7 = r = 0.  Let 

F=F  UffU£,  U = U\U,  L = L\C, 

7 = (9i~g),  T = X]  (9~9i)  where  g = , ■ 

i£U,g,>g  i£C,g,<g  ' ' 

U = U U {i  6 U : gi  > g} 

Ll+1  = Ll 

U -U\{i  : gi  > g) 


If  7 > r, 
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If  7 < r, 

L = L U {i  € C : gi  < g} 

Lt+1  = Ll 

C = C\{i  € C : gt  < g} 

F = F\({i  eU  \ gi  > g}  U {i  e C : gi  < g}) . 

Let  us  stop  this  process  if  7 = r = 0.  Then 

F = F U U U £,  U = U and  L = L 


□ 

(3)  Since  x is  piecewise  linear  with  respect  to  a,  evaluate  f\x(aj+  J), 
/'(x(a“+1)),  /'(x(0))  = /'(x(a))  |Q=0  and  /"(x(ai+1)). 

(4)  If  we  reach  the  extreme  point  of  g on  K or  some  a = aj  such  that 

g%  > g Vz  € Uj , ^ < 7 Vi  G Lj  and  gi  = g \/i  <E  Fj, 

then  stop  this  continuing  procedure  and  go  to  starting  procedure  with  the  global 
minimizer  x(a)  for  given  g as  new  starting  point  x0. 

If  old  derivative  of  f,  / (x(aj_i)),  is  less  than  0 and  derivative  of  f at  aj , 

/ (x(a~)),  is  greater  then  0,  then  local  minimum  of  / occurs  at  x(a)  where  a = 
/(V°)) 

If  / (z(a“))  < 0 and  f'(x(atj))  > 0,  or  f\x(aj))  < 0 and  f' (x(aj))  = 0,  then  local 
minimum  of  f(x(a))  occurs  at  a = 07. 

If  we  can  find  another  local  minimum  of  f(x(a))  in  the  interval  [aj,aJ+1], 
compare  this  with  the  best  value  of  f(x(a))  in  the  interval  [0,Qj],  and  then  keep 
the  smaller  value  f(x(a))  as  the  global  minimum  and  the  corresponding  x(a)  as  the 
global  minimizer  for  a given  g.  Then  repeat  this  continuing  procedure  until  meet 
stopping  point  or  extreme  point  of  g on  K . 

4 Error  Evaluation 

In  this  error  evaluation  procedure,  we  will  check  the  first  order  condition  and 
define  fj,  and  A. 
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Given  a scalar  A,  we  define  the  vector 

/}(x,  A)  = Al  — 2Ax  + Al. 

To  satisfy  the  first  order  condition,  fa  = 0 if  i £ F,  fa  < 0 if  i € U and  fa  > 0 if 
i £ L.  Hence, 


A = 


lp  (Al  - 2Ax)i 


max(Al  — 2Ax)l/ + min(Al  — 2Ax)l 


\F\ 

+ 

2 


if  \F\  ^0, 


if  |F|  = 0. 


Now  evaluate  the  error. 


Error  = error  in  F component  -f  error  in  U component  + error  in  L component 
= ||  F component  ||2  + ||  U component  ||2  + ||  L component  ||2 

— II^f+  ||  + ||Mt/+||  + ||ml+|| 

where  pF+  = {i  € F : fa  / 0},  / = {*  € U : fa  > 0}  and  p.L+  = {i  6 L : /rt-  < 0}. 

If  error  = 0,  then  it  satisfies  the  first  order  condition  and  so  go  to  perturbation 
procedure  to  check  the  second  order  conditions  and  so  on.  Otherwise,  it  is  not  a local 
minimizer,  and  so  in  the  perturbation  procrdure  we  do  not  have  to  check  the  second 
order  conditions. 

5 Perturbation  Procedure 

In  this  perturbation  procedure,  define  Upper  and  Lower  are  active  sets  and 
Free  is  the  set  of  free  indices,  i.e.,  Upper  = {i  : X{  = 1},  Lower  = {i  : Xi  = 0}  and 
Free  = {i  : 0 < x,-  < 1}  which  are  different  from  17,  L and  F,  and  check  the  following 
second  order  conditions  : 

(1)  For  each  i and  j £ Free,  a,j  = 1. 

(2)  Consider  the  three  sets  Upper0,  Lowero,  and  Free , where  Upper0  = {i  : X{  = 

1 and  pi  = 0}  and  Lowero  — {*  : = 0 and  pt  = 0}.  For  each  i and  j in  two 

different  sets,  we  have  a8j  = 1. 
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If  it  satisfies  above  second  order  conditions,  then  it  is  a local  minimizer,  if 
does  not,  it  is  a stationary  point.  If  it  is  a local  minimizer  with  no  free  index,  i.e., 
all  components  of  local  minimizer  are  0 or  1,  then  go  to  the  Kernighan-Lin  method 
for  local  minimizer.  If  it  is  a stationary  point,  then  perturb  the  components  which 
violate  the  condition.  If  it  is  a local  minimizer  but  has  free  indices,  then  perturb  free 
components  by  using  the  power  method. 

Perturb  the  point  in  order  to  at  least  one  more  component  become  boundary. 

Then  go  to  starting  procedure  with  the  perturbed  point. 

5.1  Perturbation  by  Using  Power  Method 

We  want  to  perturb  the  free  components  of  x so  that  at  least  one  free  compo- 
nent of  x become  0 or  1,  by  adding  a vector  z such  that  maxz  zT  Appz  and  lTz  = 0 
where  A ff  is  the  matrix  of  free  components.  Let  P = I — |llT  be  the  projection  to 
1 where  k is  the  size  of  App.  Then  we  notice  that  P2  = P and  PT  = P. 


Lemma  19  Let  A be  a symmetric  k x k matrix  and  let  Ai  < • • • < A*,  be  its  (real) 
eigenvalues.  Then 

Ai ||y ||2  < yT Ay  < A„||y ||2  for  all  y € Rk. 


Proof.  See  p545  in  [3].  □ 

Theorem  20  If  diag(A)  = I,  = 0 or  1 for  all  i,j  and  a{j  = 1 for  some  i ^ j, 
then 

min  xrPAPx  = min  x7  Ax,  P = I — — 11T. 

||x||=l  ||x||=l,x_Ll  k 
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Proof.  Let  ||  ■ ||  be  the  Euclidean  norm  and  ||x||  = 1.  Define  Xp  is  the  projection  of 
x on  1 and  x^  = (x  — xp)  is  perpendicular  to  1.  Then  Pxp  = 0 and  Px/v  = xyy. 


min  xtPAPx 

IMI=i 


mm 

I|xn||2+||xp|P=i 


xJjAffX-n 


min 

||x^||<l,xXl 


‘-/V 


AxN. 


Under  the  hypothesis,  all  entries  of  A are  one  or  zero  with  atJ  = 1 for  some  i / j and 
lTx  = 0.  Hence,  we  have  xTAx  — 0 with  x,  = 1,  Xj  = — 1,  and  Xi  = 0 for  l / i,j. 
By  lemma  19,  A has  eigenvalue(s)  which  is(are)  less  than  or  equal  to  zero.  We  claim 
that 


min  xlAxjv  = min  xIAxm 

||xw||<l,xll  HxArll^l.XArll 


(4.48) 


min||x*||<1,xtf.Li  x^Axn  < minux^n-^x^x!  xJjAxn  is  obvious.  Now  we  want  to  show 
the  other  inequality.  If  the  norm  of  the  solution  of  left  hand  side  in  (4.48)  is  one,  we 
have  equality.  Suppose  xN  is  the  solution  of  left  hand  side  in  (4.48)  with  ||xyv ||  < 1, 
then  there  exist  (3  > 1 such  that  ||/?xjv||  = 1.  Let  y = (3xN.  Since  (3  > 1 and 
xJjAxn  < 0, 


y1  Ay  = (32xtn  Axn  < xtnAxn. 


Thus,  we  hold  (4.48).  Since,  Pxp  = 0 and  Px^  = x^v,  we  have 


min  xtPAPx  = min  xTAx,  P = I — -llr. 

11x11  = ! Ilx||= 1 ,x_Ll  k 


□ 


Corollary  21 


min  xljAxyv  > min  xTAx. 

||xjv||=1,xjvX1  ||x||=l 


Proof.  It  is  obvious  because  left  hand  side  has  less  conditions.  □ 

Hence  by  theorem  20  and  corollary  21,  the  smallest  eigenvalue  of  PAppP  can 
be  bounded  from  below  by  the  smallest  eigenvalue  of  App  which  can  be  estimated 
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by  Gershgorin’s  theorem. 

k k 

| A — 1 1 < \ciij\  1 — \aij\  < A.  (4.49) 

Let  Xrnin  and  Amai  be  the  most  negative  and  positive  eigenvalues  of  PAp^P  respec- 
tively. 

Theorem  22  If  a > — 1— ^ , ^en  f}ie  dominant  eigenvalue  of  M = 
PAi?^P  + al  is  the  most  positive  eigenvalue  o/M. 


Proof.  We  know  by  (4.49)  that  all  eigenvalues  of  PAp^P  are  bounded  from  below 
by  1 — |a,j|.  By  Lemma  19,  we  know  that  Amin  < 0 < Amax.  Let  Am  and  A m 

be  the  most  negative  and  positive  eigenvalues  of  M respectively,  then  Xm  = Am;n  + a 
and  Xm  = Amax  + a.  Hence, 


a > 


mina(l  - Ej= ij#  M) 

2 


a > — - 


A, 


a > 


^ min  “h  ^ max 


(^mm  + a)  < A 

max  + a 

Am  T Aj^f 

| Aot  | < |Am| 


Hence  if  we  choose  a such  that  a > — then  the  dominant  eigenvalue  of  M is 
the  most  positive  eigenvalue  of  M.  □ 

Now  we  apply  a small  number  of  iterations  of  the  power  method  as  follows: 


Zk+\  = Mzfc,  M = PAjrpP  + al 

_ Zfc+l 

u llz*+i|| 

Zfc+l  = u 

Let  us  stop  this  power  method  if  ||z;t+i  — Zfc||  0.  Then  u is  the  eigenvector 


corresponding  to  the  dominant  eigenvalue  of  M.  Since  eigenvectors  for  M are  equal 
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to  the  eigenvectors  of  PA/^?P,  u is  the  eigenvector  corresponding  to  the  dominant 


eigenvalue  of  App. 

If  x whose  components  are  all  0 or  1,  does  not  satisfy  the  conditions  of  a local 
minimizer  or  0 < X{  < 1 for  some  i and  so  x is  perturbed,  then  go  to  the  starting 
procedure  with  the  perturbed  x.  If  x is  a local  minimizer  and  x,  — 0 or  1 for  all  i, 
then  go  to  the  exchange  method  for  a local  minimizer. 


Now,  we  reach  a local  minimizer  whose  components  are  all  0 or  1.  In  this 

section,  thus  we  assume  that  all  components  of  the  local  minimizer  x are  0 or  1. 

6.1  Kernighan-Lin  Method 

Let  us  see  the  Kernighan-Lin  method  (see  [13]).  Suppose  that  the  matrix  C 
is  symmetric  and  diagonal  is  zero.  Let  H = {i  : X{  = 0}  and  B = {i  : X{  = 1}.  Define 
for  each  h\  £ H,  an  external  cost  Ebl  by 


Similarly,  define  Ebl,  Ibl  for  each  bi  £ B.  Let  Dz  = Ez  - Iz  for  all  z £ S — H U B; 
Dz  is  the  difference  between  external  and  internal  costs. 

Lemma  23  (Kernighan-Lin  method)  Consider  any  h £ H,  b £ B.  If  bi  £ B 
and  hi  £ H are  exchanged,  the  gain  (that  is,  the  reduction  in  cost)  is  precisely 


6 Exchange  Method  for  a Local  Minimizer 


and  an  internal  cost  Ibl  by 


heH 


9bihi  — Dbl  Dhi  2cblh1 


(4.50) 


Furthermore,  we  can  recalculate  the  D value  by 


D1,,  = Db  - 2cbhl  + 2cbbl  for  b £ B - {bi} 
D'h  = Dh  + 2chhl  - 2chbi  for  h £ H — {hi}. 


(4.51) 


□ 
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6.2  Improved  Kernighan-Lin  Method 

In  the  Kernighan-Lin  method,  if  the  maximum  gain  is  less  than  equal  to  zero, 
then  stop  the  process,  but  we  want  to  continue  the  process  even  if  the  maximum  gain 
is  zero.  Since  even  if  the  maximum  gain  is  zero  in  current  iteration,  we  may  have 
positive  gain  in  the  next  process  with  the  exchange  of  components  whose  gain  is  zero 
in  current  iteration.  So  we  concern  all  b € B and  h £ H such  that  g^h  > 0.  We  will 
compare  the  numerical  result  of  the  Kernighan  and  improved  Kernighan-Lin  methods 
for  the  quadratic  problem  /(x)  = (1  — x)rdx.  Hence  the  formulas  of  D'(b),D\h) 
and  gb^i  for  improved  will  be  exactly  same  as  those  for  the  Kernighan-Lin  method. 
But  the  updated  sets,  newH  = H + {b\}  — {hi}  and  newB  = B + {hi}  — {&i}  for 
the  improved  Kernighan-Lin  method  are  different  from  the  sets  newH  = H — {hi} 
and  newB  — B — {&i}  for  the  Kernighan-Lin  method.  Since  we  continue  the  process 
even  if  the  maximum  gain  is  zero,  we  need  to  choose  which  components  whose  gain 
is  zero  in  the  current  iteration  will  be  exchanged.  Since  newH  U newB  is  equal  to 
H U B in  the  improved  Kernighan-Lin  method,  we  also  need  to  control  the  infinite 

loop.  We  will  see  the  details  later  in  the  section,  exchange  method  procedure. 

6.3  Formulas  for  Exchange  Method 

The  formulas  in  this  section  (4.52)  and  (4.54)  are  the  formulas  for  the  improved 
Kernighan-Lin  method  for  the  quadratic  problem  /(x)  = (1  — x)rAx. 

Lemma  24  Let  x be  an  n-vector  with  all  components  0 or  1 and  /(x)  = (1  — x)TAx. 
If  y be  an  n-vector  such  that  y\  — xk  = 1,  yk  = %i  = 0 and  yi  = xt  Vi  ^ l,k,  then  the 
gain  g = /(x)  - /( y)  is 

(n  n \ / n n \ 

^ ^ a/ci  2 ^ ^ akiXi  | T I 2 ^ ^ anXi  ^ ( an  j T 2 2 aki.  (4.52) 

i= 1 i= 1 / V i=l  i= 1 J 


Proof. 


(1  - x)tAx 
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i=l 

n 


0 = 1 


_x*) 


1 = 1 


i^l,k 


^ aijXj  + auxi  + dlkxk 

. \j^l,k  / 

^ ^ dij Xj  I “f"  CL{lXl  “f~  Q,{kXk 
\&l,k  ) 


+ (1  - Xi) 

+(1  - xk) 


y\  dijXj  j + duXi  + dikXk 
- / 

^ ' dkjXj  j ~f~  dklXl  ~h  dkkXk 

L \j^l,k  / 


X'}  ( ^ aijXJ  j + ( 

i^l}k  \j^l,k  J i^l,k 


(1  ^t)  “I"  Q'ik'Ek) 


y aijXj  J + duXi  + aikxk 

- \j^l,k 


+(1  ~ xi) 

+(1  - xk) 

Since  yi  — xk , yk  = xi  and  y,  = xt  Vz  ^ /,  fc, 
(l-y)TAy 


^ ^ Q'kj^'j  J “1“  ttkl^l  + Ukk^k 

j&,k 


^ 1 •£»)  ( ^ ^ dijXj  j *f-  ^ ^ ( 1 Xi)  (ydi\Xk  ~|-  d{kX[) 

i^l,k  \j^l,k  / i^l,k 


i^l,k  \j^l,k 

+(1  - xk) 


y aijXj  j + daxk + dikxi 

- \j&,k 


+ (1  ~ Xl) 

Hence,  the  gain,  old  cost  - new  cost  = /(x)  — /( y),  is 


y j dkjXj  j “l-  o,kixk  -j-  akkxi 

- \j¥=l,k 


g = (1  - x)tAx  - (1  - y)TAy 

= 5^(1  - Xi)  [du(x,  - Xk)  + dik(xk  - X;)] 


i^l,k 

+ (1  - Xl) 


^ ' aijXj  j ( ^ ^ dkjXj  j -j-  ((J;/  dkk)xi  [dik  dki)xk 

yj^l,k  J \j^l,k  / 
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+(1  ~ xk) 


y.  akixj  ] ( y 


^ ljxj  ] ~b  (^fc/  ^lk)X(  ~f"  (akk  ®ll)Xk 

- \j&>k  / \j^l,k 

yy  ( i x^i^d{k  Q'n')  ~t"  y ^ ^ijxj  y ^ ^ kjXj 

i^l,k  j^l,k  j^l,k 

( since  xi  = 0,  xk  = 1,  aik  = aki) 

^ ^ ( j-  2 X{^i^d{k  & n ) 

i^l,k 
n 

^(1  2 <2Z/)  (1  2a;/)  (a/*  ^//)  (1  2^/^ ) ( c^/cA; 

2 = 1 
n 

= ^(1  - 2xi)(aik  - an)  + 2 - 2akt  since  xi  = 0,  xk  = 1,  an  = akk  = 1 
2 = 1 

f n n \ / 71  n\ 

( y ' ^ ki  2 y ' ak{X{  J "f"  I 2 y ^ a 1 , x l y ' an  J -)~  2 2aki 


, i=i 


i=  1 


i=l 


»'=  1 


□ 


Let  us  define  D(i ) = J2j=ia>j  ~ 2 ^”=1  aijXj  for  all  i,  and  H — {i  : Xi  = 
0},  B = {i  : = 1}.  Let  us  define  gbh  for  h E /L,  h E B,  as  the  gain  by  exchanging 

h 6 B and  h E Lh,  that  is, 


9bh 


i n \ / n n 

y ^ ^ ki  2 y ^ ak{Xt  j “t-  I 2 y ^ anx^  y ^ rt/j  | -j-  2 2aki. 


,!=1  != 1 / \ i=l  i=l 

Lemma  25  Ifb\  E B and  hi  E H are  exchanged  i.e.  new  H = H'  = H {6X } — {hi} 
and  new  B = B = B + {hi}  — {hi},  then  for  i £ B'  and  i € 


g — -D(hi)  — D(h\)  + 2 — 2ab1hi 
and  D(i)  can  be  recalculated  by 

D(i)'  = D(i)  -f  2aib1  - 2alhl . 


(4.53) 


(4.54) 


Proof.  (4.53)  is  followed  by  (4.52)  in  Lemma  24.  Since  hi  E B and  hx  E H are 
exchanged,  x ^ = 1 and  xkl  = 0.  Hence, 


D(i)'  = 

,j=i  j= l 
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— ^ ^ d{j  2 | ^ ^ CLijXj  d{b,Xbl  4”  Q,ih\'Eh\  4”  ^ihi^bi 


i=i 

n 


0 = 1 


— ^ ^ d{j  2 ^ ^ d{j X j + 2id{bi  2tt {h\ 

j= 1 j=l 

— Z)(z)  4“  2ct;^j 


□ 


We  want  to  show  that  our  formulas  for  exchange  method  are  equivalent  to 
the  formulas  in  the  improved  Kernighan-Lin  method  for  the  quadratic  programming 
problem  /(x)  = (1  — x)TAx.  Since  the  matrix  C in  the  Kernighan-Lin  method  is 
symmetric  and  all  diagonals  are  zero,  the  relation  between  C and  our  matrix  A is 

A = C 4- 1. 


Lemma  26  For  given  conditions, 


Dh  — — D(h ) 4*  1 and  Db  = D(b)  4-  1 


(4.55) 


where  Dh  and  Db  are  in  the  Kernighan  method,  and  D(h)  and  D{h)  are  in  our 
exchange  method.i.e 

n n n n 

Dh  = 2 ^ dhjXj  - ^ ahj  4-  1 and  Db  = ^ abj  - 2 ^ ahjx3  -f-  1. 
j=i  j= i j= i j=i 


Proof.  For  h <E  //, 


Dh  — External  cost  — Internal  cost 

= Cfit  ~ y~!  c,ii 

j'es  je// 


— dhj  — j a/jj  — 1 ) since  h £ Ft  and  C — A — I 

jeB  \jeH  J 

n n 

= ahjXj  ~ ah^1  ~ + 1 


j=i 


j= i 


— 2 'y  ^ ahjXj  y ' ahj  4"  1. 
j= i i=i 
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For  b G B, 


Db  = External  cost  — Internal  cost 

= y ^ °i>j  — yi  °bj 

jeH  jeB 


y!  abj  ( yi  abj  i j 

jeH  \jeB  J 


since  b G B and  C = A — I 


= Y ab^1  ~ ~ Y abixj  + 1 

j=l  3 = 1 

n n 

= ^ ^ G6j  — 2 ^ ^ abjxj  T 1- 

j=i  j=i 


□ 


Theorem  27  Under  above  conditions,  the  formulas  (4.52)  and  (4.50)  are  equivalent, 
and  the  formula  (4.54)  is  equivalent  to  the  formula  (4.51). 


Proof.  Since  h\  G H and  bx  G B are  exchanged,  by  (4.53)  in  lemma  25  and  (4.55) 
in  lemma  26, 

gbihi  = D(b)  - D(h)  + 2 - 2 ablhl 

^ 9bih\  = Db\  T Dh\  /i, 

gblhl  = Dbl  + Dhl  - 2 cblhl  since  bx±hx. 

This  shows  that  (4.52)  is  equivalent  to  (4.50).  Now  Let  us  show  that  new  Db,  D'h  = 
Dh  + 2chh,  - 2 chbl  and  new  Db,  D'b  = Db  + 2 cbbi  - 2 cbhl  where  h G H - {hx}  and 
b G B — {61}  are  equivalent  to  D(i)'  = D(i)  + 2a,q  — 2at/n. 

For  h G H—{hx}  or  h G H-\-{bx}  — {hx},  and  b G B — {bx}  or  b G 5+{/zi}  — {^i}, 


Hence, 


Ubhi  Q)/ii  ? ^66i  Q>6i  ? ^hhi  ^hh\  •>  ^hb\  ^/i6i  • 


Db  = Db  — 2 Cbh,  + 2c66, 
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44"  + 1)  — D(b)  + 1 — 2 Cbhl  + 2cbbl 

44  .0(6)  + 1 = 0(6)  + 1 — 2a,bh1  + 2 abb! 

44  0(6/  = 0(6)  - 2abhl  + 2a6fcl. 

Similarly,  we  have 

Dh  = Dh-  2 chhl  + 2 chbl  44  0(6)  = 0(6)  - 2 ahhi  + 2 ahbl. 

Hence,  (4.54)  = (4.51).  □ 

Thus,  the  formula  for  gain  in  our  exchange  method  is  equivalent  to  the  formula 

for  gain  in  the  Kernighan-Lin  method. 

6.4  Exchange  Method  Procedure 

We  will  start  with  H = {i  : X{  = 0}  and  B = {i  : X{  = 1}  and  separate  two 

stages. 

First  stage.  we  compute  D(j)  for  all  j and  find  maximum  of  0(6)  for 
6 £ B and  minimum  of  D(h)  for  h £ H.  Let  D(b*)  = ma x{0(6)  : b £ B}  and 
D(h*)  = min{0(6)  : h £ H}. 

Lemma  28  If  D(b*)  - D(h*)  + 2 < 0,  then  gbh  < 0 for  all  (6,  h)  £ (O,  H). 

Proof.  Suppose  0(6*)  — 0(6*)  + 2 < 0.  For  any  b £ B and  h £ H,  since  D(b)  < D(b*) 
and  0(6)  > 0(6*), 

D(b)  - D(h)  + 2 < D(b *)  - 0(6*)  + 2 < 0. 

Since  abh  = 0 or  1 V(6,  h)  £ ( O,  H), 

D(b)  — 0(6)  + 2 < 0 =4  gbh  — D(b)  — D(h)  + 2 — 2abh  < 0. 

□ 

Hence,  if  0(6*)  — 0(6*)  + 2 < 0,  then  all  gain  is  negative  and  so  we  have  to 


stop. 
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Theorem  29  For  b E B,  if  D(b)  < D(h*)  — 2 then  for  all  h E H , gbh  < 0.  Also  for 
h E H , if  D(h)  > D(b*)  + 2 then  for  all  b E B , gbh  < 0. 

Proof.  Suppose  b E B is  given  and  D{b)  < D(h*)  — 2.  Then  D(b)  — D(h*)  + 2 < 0 
and  so  for  all  h & H, 

gbh  = D(b)  — D(h)  + 2 — 2abh  P D(b)  — D(h  ) + 2 — 2 < 0. 

Similarly,  for  given  h 6 H,  if  D(h)  > D(b*)  + 2 then  for  all  b 6 5,  gbh  <0.  □ 

Hence  we  consider  only  {6  6 5 : Z)(6)  > D(h*)  — 2}  and  {h  E H : D(h)  < 

D{b*)  + 2}. 

Theorem  30  If  D(b *)  < D(b*)  — 2 for  some  6*  6 B or  D(h *)  > D{h*)  + 2 for  some 
h * £ H , then  gb,h  < gb*h * and  gbh,  < gb*h»  for  all  b E B and  h E H . 

Proof.  Suppose  D(b,)  < D{b*)  — 2,  then  for  all  h E H, 

9b,h  = D{K)  - D{h)  + 2 - 2abth 

< D(b*)  — 2 — D(h)  + 2 — 2ab,h 

< D(b*)  — D(h) 

< D(b*)  — D(h)  + 2 — a,b*h* 

= gb* h>  ■ 

Similarly,  if  D(h *)  > D(h*)  + 2,  then  for  all  b E B,  gbh,  < gb*h* ■ □ 

Hence,  we  do  not  consider  b E B such  that  D(b)  < D(b*)  — 2 and  h E H 

such  that  D(h)  > D(h*)  + 2.  That  is,  we  consider  only  {b  E B : D(b)  > D(b*)  — 2} 

and  {h  E H : D(h)  < D(h*)  + 2}  to  be  exchanged.  Therefore,  by  theorem  29  and 
theorem  30,  we  just  consider 


{b  E B : D(b)  > D(b*)  - 2,  D(b)  > D(h *)  - 2} 
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and 

{heH:  D(h)  < D(h*)  + 2,  D(h)  < D(b *)  + 2}. 

Let  us  define  for  i = 0,1,2, 

Bi  = {b  G B : D(b)  = D(b*)  - *,  D(b)  > D(h*)  - 2} 

and 

H{  = {he  H : Z?(/i)  = F(/i*)  + t,  D(/i)  < D(6*)  + 2}. 

Lemma  31  For  (6,-,/jj)  € (Bx,Hj),  if  i + j > 2,  then  gboho  > gb.hj  where  (b0,h0)  G 
(Bo,  H0). 


Proof.  Suppose  i + j >2  and  ( bi,hj ) G ( Bi,Hj ).  Then  since  = 0 or  1 Vi,  j, 

<760/io  — D(bo)  — D(ho)  + 2 — 2ab0h0 

= F(6i)  + i — (D(hj)  — j)  + 2 — 2a,b0h0 

> D(bi)  + i - D(hj)  + j 

> D{bi)-D(hj)  + 2 

> D(b{)  — D(hj)  + 2 — 2abthj 


If  the  maximum  gain  is  positive,  then  select  and  exchange  any  pair  of  compo- 
nents which  makes  maximum  gain  and  then  go  back  to  first  stage  with  recalculated 
D(i ) by  using  (4.54).  If  the  maximum  gain  is  zero,  then  go  to  the  second  stage  with 
the  pairs  of  components  whose  gain  is  zero,  say  ( boz,hoz ),  to  make  better  selection 
for  next  iteration. 


Second  stage.  In  this  stage,  everything  is  temporary  i.e.  we  will  not  bring 
any  value  in  this  stage  to  next  iteration.  We  just  check  which  pair  of  components  of 
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zero  gain  can  make  positive  gain  in  the  next  iteration.  So,  for  efficiency  of  program, 
we  will  consider  only  B0  U U B2  and  H0  U Hx  U H2  as  B and  H.  In  this  stage,  to 
choose  which  pair  of  components  ( b0z,hoz ) whose  gain  is  zero  in  a current  iteration 
will  be  exchanged,  we  assume  for  each  pair  {boz,h0z),  b0z  and  h0z  are  exchanged  and 
then  calculate  the  temporary  D(i),  for  i in  Hr  = Uf_0fT  U {6o2}  — {ho2}  and 

Bt  = U ?=0Bi  U {/i0z}  - {b0z}  by  (4.54). 

We  are  just  looking  for  the  largest  possible  positive  gain.  We  call  the  positive 
gain  in  this  stage  possible  positive  gain,  because  it  is  not  really  positive  gain  in 
this  iteration  but  the  exchange  of  boz  and  hoz  will  produce  positive  gain  in  the  next 
iteration.  Define 

Dr(b*)  = max{Z)7’(6)  : b £ B}  and  Dj(h*)  — min {Dr(h)  : h £ H}. 

Then  similarly  to  the  first  stage,  if  D(b*)  < Dr(h*)  — 2,  then  stop  and  so,  for  f = 0,1, 
define 

Bt,  = {b  £ Bt  '■  DT{b)  = DT{b*)  — i,  DT(b ) > Dt’(/j*)  — 2} 

and 

Ht,  = {h  € Ht  Dt(/i)  = Dr(h*)  + i,  Dt(B)  < DT(b*)  T 2}. 

For  some  (boz,hoz),  if  we  find  a possible  positive  new  gain,  then  exchange  the  pair 
of  components  (boz,hoz)  and  then  go  back  to  first  stage.  If  (boz,hoz)  has  no  possible 
positive  gain,  then  do  it  again  with  (b0z+1 , h0z+l ) until  2 is  exhausted.  If  there  is  no 
positive  gain  for  all  of  the  pair  of  components, (6oz,  hoz),  then  stop  the  process. 

But  in  practice,  even  if  maximum  gain  is  zero  and  for  all  (boz,hoz),  there  is 
no  possible  positive  gain  in  a current  iteration,  we  will  continue  the  iterations  by 
exchanging  anyone  of  (b0z,  hoz)’s.  Because  under  the  above  circumstance,  we  may 
have  some  positive  in  a later  iteration.  Then  we  have  to  consider  an  infinite  loop. 

Lemma  32  If  there  is  a cycle,  then  for  each  iteration  in  the  cycle,  the  maximum 
gain  is  zero. 
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Proof.  Suppose  that  (l  — x^)1  Ax^p*  = (l  — x^)1  Ax ^ with  q > p.  Since  we  do 
not  take  a negative  gain,  for  each  j = p + 1,  • • • , q, 

(l  — x(j-1))7  Ax^-1*  > (l  — x^)T  Ax^. 

Hence, 

(1  - x(p))T  Ax(p)  > • • • > (1  - x(p+1))T  Ax(p+1)  > • • • > (1  - x(9))7’  Ax(?). 
Thus, 

(l  — xW-1))7  Ax*J_1*  = (l  — x^)7  Ax^J*  Vj  = p + \,p  -f-  2,  • • • , q. 

Therefore,  every  gain  from  x^  to  x^  is  zero.  □ 

Hence,  to  avoid  infinite  procedure,  we  need  to  stop  if  we  have  gain  zero  con- 
secutively for  a while. 

Note  in  Figure  4.4  that  D(b*)  denotes  maximum  of  D(b)  for  b £ B,  D(h*) 
denotes  minimum  of  D(h)  for  h £ H and  g{b,h)  = D(b)  — D(h)  + 2 — 2abh  for  all 
b £ B and  h £ H.  Dr{j ) is  the  value  of  D(j)  for  j E B*  = U ]=0Bt  and  H*  = Uj=0Hi 
after  changing  two  nodes.  In  practice,  even  if  z is  exhausted  in  the  last  step,  we  can 
go  back  to  the  beginning  of  this  procedure  with  exchanging  last  pair  of  zero  gain 
components  (bo^h0z)  until  it  happen  for  a few  times  consecutively. 

7 Stopping  Criteria  for  a Local  Minimizer  after  the  Exchange  Method 

In  the  Exchange  Method  Procedure,  if  no  components  of  the  point  is  changed, 
then  it  is  a local  minimizer  and  so  stop  the  program.  If  it  is  changed,,  then  we  need 
to  check  that  whether  it  is  a local  minimizer  or  not.  So,  we  will  go  back  to  starting 
procedure. 

Finally,  we  finished  the  whole  processes  to  partition  the  given  graph  into  two 
sets.  Now  we  consider  some  improvement  by  exchanging  sets  instead  of  exchanging 
nodes,  one  from  each  partitioned  graph. 


Figure  4.4:  Exchange  method  for  a local  minimize!'. 
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8 Exchange  the  Subsets  from  Partitioned  Graph 

We  partitioned  an  initial  graph  into  two  subgraphs,  V and  W.  In  this  phase,  we 
now  exchange  some  components  of  V and  those  of  W to  reduce  the  edge-cut  between 
V and  W.  That  is,  we  can  reduce  the  edge-cut  between  V and  W by  swapping  subsets 
Vi  of  V and  W\  of  W.  Let 

V = Vi  U V2,  Vi  n V2  = 0,  W = Wi  U W2  and  W1nW2  = V) 

where  V\  = {i  : xXt  = 1},  W\  = {i  : x2t  = 1},  V U W = {1, 2,  • • • , n}  and  V D W = 0. 

If  the  edge-cut  between  Vi  and  V2,  and  between  W\  and  W2  is  less  than 
that  between  V\  and  W2,  and  between  V2  and  Wi,  then  the  edge-cut  between  V 
and  W will  be  reduced  by  exchanging  V\  and  W\  where  Vt  C V,  W\  C W and 
|Vit  = \WX\.  We  use  the  gradient  projection  method  with  active  set  strategy  as 
before.  The  implementation  is  almost  the  same  as  before. 


CHAPTER  5 
NUMERICAL  RESULT 


At  first,  we  test  our  scheme  on  a small  graph  (see  Figure  5.1  ) with  the  initial 
point  x0  = 1T^  to  see  the  improvement  of  exchange  method. 

In  Table  5.1  we  will  see  the  numerical  result  of  Kernighan’s  method  and  ex- 
change method  (improved  Kernighan’s  method)  for  the  quadratic  programming  prob- 
lem 

min /(x),  /(x)  = (1  - x)TAx 

subject  to  0 < x < 1,  l7x  = m 

with  random  generated  initial  point,  x0  with  all  0 or  1 components. 

In  Table  5.1,  note  that  n(g  = 0)  denotes  the  maximum  number  of  consecutive 
zero  gain  steps  allowed,  where  n(g  — 0)  = 0 implies  only  positive  gain  is  accepted, 
i.e.,  Kernighan’s  method. 

From  Table  5.1,  we  can  see  that  there  is  a big  improvement  between  Kernighan’s 
method  and  exchange  method  (improved  Kernighan’s  method),  but  for  the  exchange 
method,  there  is  not  much  difference  between  2 consecutive  zero  gain  steps  and  4 
consecutive  zero  gain  steps. 

In  Table  5.2  with  the  initial  point  xo  = 1T~,  we  compare  the  numerical  result 
of 

(i)  the  gradient  projection  method,  and 

(ii)  the  combination  of  (i)  and  exchange  method  for  a local  minimizer  whose  com- 
ponents are  all  0 or  1 with  up  to  2 consecutive  zero  gain  steps. 

In  Table  5.2  note  that  G.P.M.  and  E.M.  denote  respectively  the  gradient 
projection  method  and  the  exchange  method  (improved  Kernighan’s  method). 
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Figure  5.1:  Example  of  graph  with  20  vertices 
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Table  5.1:  Comparison  of  Kernighan’s  method  and  the  exchange  method. 


m 

f(x)  \ n(g=0) 

0 

2 

4 

2 

3 

885 

1000 

1000 

4 

115 

3 

5 

874 

1000 

1000 

7 

126 

4 

7 

592 

850 

850 

8 

212 

9 

196 

150 

150 

5 

9 

597 

817 

817 

11 

325 

183 

183 

12 

69 

16 

9 

6 

10 

586 

666 

671 

13 

204 

288 

288 

14 

116 

5 

15 

39 

16 

44 

41 

41 

17 

11 

7 

11 

561 

629 

629 

14 

104 

144 

144 

15 

212 

227 

227 

16 

101 

17 

21 

18 

1 

m 

f(x)  \ n(g=0) 

0 

2 

4 

8 

11 

345 

477 

477 

13 

70 

70 

70 

14 

198 

352 

352 

15 

93 

16 

189 

69 

69 

17 

64 

32 

32 

18 

31 

19 

10 

9 

13 

481 

559 

559 

14 

308 

407 

423 

16 

13 

17 

114 

13 

13 

18 

61 

21 

5 

19 

13 

10 

13 

212 

804 

804 

14 

390 

14 

390 

15 

161 

196 

196 

17 

77 

18 

134 

19 

26 
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Table  5.2:  Result  of  the  combination  of  the  gradient  projection  method,  and  the 
exchange  method  with  the  initial  point  x = 1 • 


m 

global  min. 

G.P.M 

G.P.M.  & E.M. 

2 

3 

3 

3 

3 

5 

5 

5 

4 

7 

7 

7 

5 

9 

9 

9 

6 

10 

10 

10 

7 

11 

12 

11 

8 

11 

13 

11 

9 

13 

19 

14 

10 

13 

14 

13 

In  the  Table  5.2,  we  can  recognize  that  by  using  exchange  method  we  have  a 
lot  better  result.  The  significant  improvement  is  for  all  m except  m — 9,  we  got  the 
global  minimum. 

Now,  we  test  our  scheme  on  reasonably  big  problems,  linear  programming  and 
G-sets  ( ftp://dollar.biz.uiowa.edu/pub/yyye/Gset/  ) with  the  initial  point  which  is 
the  projection  of  the  solution  of  the  ball  onto  the  convex  set  K.  In  the  Tables  5.3 
and  5.4,  we  put  for  the  better  partition  i.e.,  for  less  edge-cut  between  two  parts. 

In  Table  5.3,  we  can  see  the  result  of  perfectly  balanced  bisection  of  our 
Quadratic  Program  and  Metis  for  graph  partitioning  problems  associated  with  linear 
programming.  Note  that  the  adjacency  matrix  is  nonzero  pattern  for  AA1  where  A 
is  the  coefficient  matrix  for  the  linear  program. 

In  Table  5.4,  we  have  the  result  of  perfectly  balanced  bisection  of  our  Quadratic 
Program  and  Metis  for  G-set  graphs. 
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Table  5.3:  LP  graph  bisection  test  problems. 


Problem 

Vertices 

Edges 

QP 

Metis 

Adlittle 

56 

328 

88 

88 

Aflro 

27 

63 

9 

9 

Beaconfd 

173 

2669 

659 

★ 

825 

Vtp-base 

198 

1544 

163 

163 

Scorpion 

388 

1713 

22 

* 

23 

Degen2 

444 

6855 

1391 

~k 

1445 

fffKBOO 

524 

10091 

1183 

* 

1249 

Finnis 

497 

2771 

142 

* 

217 

Ganges 

1309 

7656 

118 

* 

GO 

Perold 

625 

6303 

423 

423 

Cycle 

1903 

27714 

133 

122  * 
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Table  5.4:  G-Set  graph  bisection  test  problems. 


Problem 

Vertices 

Edges 

QP 

Metis 

G1 

800 

19176 

7653 

~k 

7754 

G3 

800 

19176 

7609 

■k 

7746 

G4 

800 

19176 

7647 

k 

7691 

G5 

800 

19176 

7632 

~k 

7682 

G14 

800 

4694 

1129 

k 

1146 

G15 

800 

4661 

1140 

k 

1159 

G16 

800 

4672 

1121 

k 

1126 

G17 

800 

4667 

1089 

k 

1139 

G51 

1000 

5909 

1395 

~k 

1427 

G52 

1000 

5916 

1406 

k 

1422 

G53 

1000 

5914 

1431 

k 

1457 

G54 

1000 

5916 

1399 

k 

1419 

G43 

1000 

9990 

3387 

k 

3395 

G44 

1000 

9990 

3409 

k 

3443 

G45 

1000 

9990 

3418 

k 

3460 

G46 

1000 

9990 

3380 

k 

3461 

G47 

1000 

9990 

3373 

k 

3455 

G35 

2000 

11778 

2878 

2845 

' k 

G36 

2000 

11766 

2919 

2846 

k 

G37 

2000 

11785 

2882 

2879 

k 

G38 

2000 

11779 

2899 

k 

2902 

G22 

2000 

19990 

6843 

-k 

6867 

G23 

2000 

19990 

6806 

k 

6862 

G24 

2000 

19990 

6835 

k 

6837 

G25 

2000 

19990 

6823 

kr 

6886 

G26 

2000 

19990 

6833 

k 

6887 

CHAPTER  6 
CONCLUSION 


We  have  found  that  the  choice  of  initial  guess  plays  an  important  role  for  the 
convergence  to  a local  minimizer.  In  comparing  our  graph  partitioning  algorithm 
to  the  Metis  package,  the  quality  of  min-cuts  of  our  algorithm  is  better  than  that 
of  Metis.  The  results  presented  here  are  relevant  for  the  partitioning  into  two  sets 
with  unit  weight  of  edges  and  vertices.  We  are  considering  an  extension  to  weighted 
problems. 
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